Comparing Academic Research to Industry Research
In the digital forensics & incident response industry, the usual means of research dissemination is through conference presentations. These presentations are usually 45 to 60 minutes in length, and the only documentation produced is PowerPoint files – many of which are not made available after the conference. This method of knowledge and research effort distribution makes it difficult for those who do not attend the talk to gain full value from it. To remedy this issue, a handful of conferences now record the video of speakers and post them online after the event.
While this is certainly more useful for those who do not attend the talk live, it still leaves much to be desired. In particular, many research presentations simply highlight the results of the effort and how they can be transitioned to the field. There is generally no discussion of the process that the presenter went through to perform their research, the statistical significance of the results, or the data set used to test the validity of the results. This makes the effort non-repeatable by other researchers and weakens the effect of the research. Industry research is also usually focused on particular versions of malware or operating systems instead of a more general approach that can be widely applied outside the presenter’s test environment.
Paper Submissions
This approach is in great contrast to academic conferences. When submitting to academic conferences, the first material submitted is an 8-12 page paper that undergoes peer review. This paper describes in full detail the research approach taken, how it improves upon previous efforts, where the effort fell short, the environment used to test the research, and the improvements still needed to make it applicable to use in the field, if any. Only after a paper is accepted for publication is an associated presentation developed.
The requirement of a paper submission generally leads to higher quality research as the research project must be completed or nearly completed before being submitted. This means that there is little room for “hand waving” by the author during the submission process and that only verified results are discussed. With the notable exceptions of Black Hat and DefCon, industry conferences do not enforce the inclusion of a paper with all research submissions, and as a result, many subpar presentations are accepted.
Double-Blind Peer Review
Along with the paper submission requirement, academic conferences also have the advantage of the review process being "blind”. To quote the Elsevier review guidelines document, double blind “means that both the reviewer and author identities are concealed from the reviewers, and vice versa, throughout the review process”. This again raises the quality of accepted papers as people are not accepted based on name or company recognition, which is an issue that plagues some industry conferences, but instead on technical merit.
Furthermore, the blind review process allows for honest and direct feedback, which is often muted or not possible if the reviewer is a colleague of the author(s) or if the authors know all of the reviewers. To work around this issue, academic conferences combine the blind review process along with much larger review teams than typically used for industry events. To ensure that reviewers are not assigned papers to review from their colleagues or friends, each conference generally has one or two people who assign all of the review tasks while keeping potential conflicts in mind.
Detailed Submission Feedback
Since reviewers are free from potential conflicts of interest when reviewing, this allows for detailed feedback, both positive and negative, to submissions. As someone who has had papers rejected from academic conferences, I can assure you that reviewers do not hold back with their criticism or praise. For the conferences that I have submitted to, it seems normal to receive detailed feedback from 4 to 8 reviewers along with their selected review score. Authors of accepted papers are expected to incorporate the feedback from reviews into the final versions of their paper. This process again raises the quality of published papers.
The reception of detailed feedback is in contrast to most industry conferences that simply send a standard email informing the submitter that their work was either accepted or rejected. The lack of feedback to accepted submitters provides no direction on improvements that can be made, and the lack of feedback to rejected submitters can be highly frustrating.
Benefits of Industry Collaboration with Academia
Beyond providing a venue for thorough and peer-reviewed research to be published, academic conferences also provide a number of immediate benefits to industry organizations and individuals who take advantage of them.
Building Employee Candidate Pipelines
By attending and participating in academic security conferences, a company and its employees gain immediate visibility throughout the academic community. This allows for building relationships with professors who teach forensics and security inside of programs, such as Computer Science, as well as networking opportunities with students who are engaged enough to attend conferences. These types of students generally make great future hires, and everyone in the industry is painfully aware of how hard hiring in information security is. By building relationships with professors, companies can also make hiring much easier as professors are eager to see their students become employed in a meaningful job after graduation. Besides careers for graduated students, this pipeline can also be a feeder for robust internship programs.
For my many industry friends who are adjunct professors at universities, publishing at academic conferences with your students is a great way to push them beyond what the class minimally requires and is also a great way to find future hires.
Collaboration with Students and Research Labs
Unlike industry conferences, where most research is performed and presented by a single person, academic research is often conducted in groups of 2 to 5 people. This allows for industry researchers to embed themselves within existing university research teams in order to perform large research projects and achieve results otherwise not attainable. Such partnerships can lead to business ventures between academia and industry, as well as further develop relationships with professors and students.
Influencing Curriculum Development
A constant complaint from members of the DFIR industry is that students in Computer Science and other related programs are not being offered digital forensics and computer security courses that match real-world needs. By networking with professors and students, industry practitioners can begin to influence curriculum development by showing the industry’s needs.
Why DFRWS?
Even when considering other top-tier academic conferences, such as USENIX Security, ACSAC, and IEEE S&P, I still believe that DFRWS is the best conference for collaboration between industry and academia. I hold this belief for a few reasons:
1) DFRWS has been the venue for the release of practical and highly impactful research in the malware, memory, disk, and network forensics spaces. This can be seen in Brendan Dolan-Gavitt’s work related to VADs and the registry in memory, Andreas Schuster’s work related to pool scanning and event logs, file carving, registry forensics, and memory acquisition. If you have ever used Scalpel, Volatility, Bulk Extractor, and/or the Sleuthkit then you are using tools built in part from research originally presented at DFRWS.
2) The yearly DFRWS challenges have led to ground-breaking research in memory and network forensics.
3) DFRWS already strives to mix purely academic research with research that is applicable in the field. This leads to papers with the benefits of academic research as described earlier as well immediate application to our daily jobs in the field. The existing efforts of DFRWS to bring together academic and industry researchers has already yielded significant results over the last 10+ years and I hope to expand that collaboration in my new role with the conference.
4) Student scholarships! DFRWS also provides conference scholarships for select students who present original research. The details are spelled out on the website, but this can be a nice way for students without a travel budget to offset costs.
More Reasons to Submit!
Personal Branding and Career Potential
Beyond presenting at industry conferences, being able to list peer-reviewed, academic publications on your resume is a huge career boost. For positions, such as CISO, CTO, or Director of Research, many organizations require such publications. These publications can also go a long way in justifying the N years of experience that many positions require for those without a traditional four-year degree. Also, if you ever decide to go back to school for a Masters or PhD, then you will need to demonstrate some level of research competence.
Great Location and a New Experience
If you have made it this far, and I still have failed to convince you that DFRWS is worth a chance, then what about the fact that it is in Austin this year? Even if all else fails and you hate the conference, then at least you are surrounded by great BBQ, music, and beer! DFRWS also rotates its location yearly, so by putting DFRWS on your annual calendar then you will get to visit many great cities along with learning about a bunch of cutting-edge topics in digital forensics.
Closing Thoughts
In closing, I hope that my industry colleagues will consider starting a research project and submitting the results to DFRWS. Also, don’t be intimated by the idea of submitting to an academic conference. The DFRWS archives has all of the conference’s previously accepted papers, which you can use as templates. Furthermore, for papers with strong technical merit, but that need some editing love, DFRWS provides a “shepherding” process where reviewers help you format and mold content to make your paper as well written as it can be.
If you have any questions about DFRWS, submitting, or how to shape your research ideas then please contact me.
I hope to see you in Austin next summer!
No comments:
Post a Comment