On February 17, 2021 TrustedSec hosted an 'Ask Me Anything' on our Slack Workplace with TrustedSec's Incident Response Team. Many great questions were asked and lots of information exchanged that we didn't want to get lost with time, so we've put together this blog with questions and the conversation that blossomed from them. Please note: these responses are conversational and have not been changed or edited from their original source for this blog posting.

X: Hello everyone, and thanks for hosting this! For incidents in the cloud (e.g. AWS), how do you go about capturing memory from virtual machines?

Tyler Hudak (TrustedSec): For VMs in the cloud, we go about the same way we do for physical systems. Our triage toolkit is run locally as an admin and captures the RAM using winpmem. If its a service in the cloud type deal (like heroku, etc.) then its not possible to get RAM.

X: Thanks. That's what I assumed. So you attach a new volume with the tools and another for evidence capture, or something else?

Tyler Hudak (TrustedSec): When possible. Here is a presentation we did a few months back for doing IR in the cloud. It needs updated as some things have changed, but when possible I def prefer to create a new volume that has the capturing tools on it, mount it in the VM, run the tools, then download the volume (or the files from the volume) https://slack-files.com/T0108FLRY7M-F01NTGMH1G9-d33cafbb26

Nick Gilberti (TrustedSec): Depending on the scenario, tools can also be downloaded directly (depends on opsec concerns too)

X: what kind of open source tools do you guys like to use for IR that others can implement in their tool bag?

Justin Vaicaro (TrustedSec): Eric Zimmerman tools for sure

Leo Bastidas (TrustedSec): I love me some Zimmerman and BriMor Labs triage tools

Nick Gilberti (TrustedSec): +1 Eric Zimmerman tools. Also elastic stack when appropriate. And of course TSCopy 

Leo Bastidas (TrustedSec): Also Volatility, but they had recent license changes, I believe they are copy left now, so now I'm currently looking at Rekall

Justin Vaicaro (TrustedSec): NirSoft tools too. also i am big fan of live forensic frameworks. https://www.dfir.training/tools-sw-hw. that is a super helpful site

X: Are you guys using any automation / SOAR for collecting forensic data?

Leo Bastidas (TrustedSec): We use homegrown scripts in everything from Python, Go and even Perl, but nothing like a SOAR currently

Justin Vaicaro (TrustedSec): We use our preference of local SIEMs for the most part

Tyler Hudak (TrustedSec): We are typically dependent upon what clients have to work with, so unfortunately have to plan for the lowest common denominator.

Justin Vaicaro (TrustedSec): whether Splunk, ELK, etc

Nick Gilberti (TrustedSec): We do have agent-based capabilities. But like the others said, we rely primarily on a home grown triage tool and any SIEM-type stuff we do is local and custom

X: Got it, how about terraform scripts for collecting data and uploading them to S3 for example?

Nick Gilberti (TrustedSec): I sometimes use a custom winlogbeat to ship offline Windows Event Logs into elastic stack

Justin Vaicaro (TrustedSec): so basically we take our live forensic tool output and parse it locally for the most part

Leo Bastidas (TrustedSec): I am a big fan of Resilient when @Justin Vaicaro (TrustedSec) and I worked on an engagement

Justin Vaicaro (TrustedSec): we are working on an aggregated backend parser though..in the work once you inventory your security stack you can identify where you can automate responses and dashboard building without spending anything for SOAR functionality initially

X: Another Q from my side, did you have a chance to perform IR on any SAAS apps? like Okta, SalesForce, etc? Any tips on those?

Nick Gilberti (TrustedSec): imo it comes down to identifying the logging and audit capabilities within each of those services

Tyler Hudak (TrustedSec): Exactly what @Nick Gilberti (TrustedSec) said. TBH, its not fun. You are very much limited to the logs they can provide you. Some are better than others. My recommendation is that if you do use those types of services, look NOW to see what logs are on, what can be turned on and look at it from the perspective of what information would you need in case there was a breach of data (or suspected breach of data) in that service

X: That's exactly the problem, my go to are the logs and sending them to Elastic where you try to make sense in all of this.

Tyler Hudak (TrustedSec): we've dealt with that. Thats when we let legal counsel take over. One thing I would recommend doing is looking to see how the cloud vendor is set up.

X: Do you have any recommendation on getting Sysmon off remote laptops now that everyone is WFH?

Nick Gilberti (TrustedSec): I believe most of the log shipping utilities support encrypted transport. I'm partial to winlogbeat

Justin Vaicaro (TrustedSec): Yeah it is going to come down to on-net connectivity for the remote machines to receive those logs. Take a small data set of devices and test to see what the network impact is with this log forwarding.

Leo Bastidas (TrustedSec): You can even set up a WEC with a Windows Server and use HTTPS to push native and Sysmon logs via WEF. https://channel9.msdn.com/Events/Ignite/Australia-2015/INF327

X: WEC/WEF or Winlogbeat (via Graylog Sidecar) was my plan but was curious if there was a better way

X: I am exploring the evtxtoelk parser for mass event log ingest across many instances for that common thread of lateral movement with offline images or in cases where sparse data is not ingested into siem . Do you find that approaching threat hunting of the incident scope for that master timeline to be more manageable via a common incident view? I imagine for some siem solutions, ingesting in mass to the default siem can be cost prohibitive ;)

Justin Vaicaro (TrustedSec): I worked on a threat hunt recently where the client had their servers forward logging to Splunk and user machines Sysmon and native win logs to ELK. from a design perspective this worked out very well saving on the backend infra cost (Splunk specifically)

Leo Bastidas (TrustedSec): Exactly what Justin said. Data retention cost will always be a factor when it comes to the pricing model of say Splunk and the other big SIEM vendors. I would do a cost/time analysis on if having two (or more) different products instead of one central plane of glass is worth the saving. Will the analysis ignore the other platforms? Will respond time suffer? Will alerts and data be missed?

X: I'm teaching a Forensics class (last minute and I have to rebuild the content). We have access to AWS Educate (most services, no Elastic and no SSM). Other than traditional EC2 forensics, what should I definitely cover? Already got them using VPC flow logs and Traffic Mirroring because they are quick and easy. What else would you definitely add? (e.g. CloudTrail, CloudWatch, GuardDuty, Config, WAF)

Tyler Hudak (TrustedSec): Def Cloud{Trail,Watch}. Those are definitely helpful for investigations. I can't remember, but also being able to look at billing spikes can sometimes point to areas of activity.ie. Billing spike due to excessive usage might show you a time frame what things were happening.

X: Billing spikes is a great idea. I had forgotten about that. I don't think we get to see billing, but it's worth mentioning. Thanks!

Tyler Hudak (TrustedSec): Ya its not usually something everyone has visibility into, but IIRC (at least on azure) you can set up alerts for them. Might not be a bad idea to set up an alert that goes to a bunch of people to at least use for detection

X: Hey guys!   Do you have recommendations or best practices for handling evidence gathered during an incident?  And how are those artifacts used (if at all) for improving your security posture?

Nick Gilberti (TrustedSec): Are you asking from a collection techniques perspective, or from a chain of custody perspective?

X: From a collections techniques perspective.   For example - collecting a document or memory samples found during an investigation.   How do you take that evidence, store it, save it, use it for hunting?

Justin Vaicaro (TrustedSec): Artifact IOCs can easily translate to security recommendations for containment, eradication and future security stack hardening

X: I was thinking that a threat intelligence platform was needed…maybe not

Justin Vaicaro (TrustedSec): depends what you mean by threat intelligence?

Nick Gilberti (TrustedSec): Gotcha. Well much of that will depend on the tooling you have in place. For example, if you have an EDR solution, most evidence will be stored and available in your dashboard. If not, you will need to have a collection tool available that has capabilities for raw copying locked files, maybe hashing the evidence, etc. As far as using for hunting, it comes down to identifying the indicators found in the evidence and using your available monitoring solution(s) to search for them

Justin Vaicaro (TrustedSec): true counter threat intel monitoring is different than basic threat feeds. I would suggest evaluating your use cases to help answer this question

Leo Bastidas (TrustedSec): Hey X look at something like TheHive Project for a homegrown if I'm understanding your question correctly. https://thehive-project.org/

X: Thoughts on using legal firms as intermediaries for investigations? That has been an approach to protect findings from discovery (and get lawyers paid more!) but the Capital One case, though fact-specific, limited those protections.

Tyler Hudak (TrustedSec): LOL - so we deal with lawyers all the time. I love working with them bc it means our reports are shorter (bc they don't want alot in them) As far as using it to protect discoveries, like you said, the C1 case is probably going to change that (IANAL). However, they'll continue to be used. Its not a bad thing either in a lot of cases. For example, we often get into talks with clients where they ask if they need to report the breach. Thats not our call, thats their lawyer's call. If the lawyer is already involved, it makes things a lot easier.

X: 100% agree on the reporting part (I don't want to deal with that!)

Leo Bastidas (TrustedSec): Be advise on using legal firms during investigations, I don't have the case reference off-hand, but sometimes it works as working product and attorney/client, most of the time it doesn't. This is just talking with peers

X: This is often overlooked. Most of the time, even the way the communications and/or business relationships are set up will break those privileges and it's admissible anyways. (Personally, I'm glad the C1 case brought that to light. Lawyers are important and should be involved, but they also want to up billable hours...)

X: Any thoughts on EDR solutions? As in, which ones seemed to be subverted the easiest...

Justin Vaicaro (TrustedSec): hmm, i am not going to address a specific vendor product, but from my past i built a series of use cases and then tested

Tyler Hudak (TrustedSec): You'd have to ask our research team on that. Honestly, we see all types of EDR and they all provide some value in one form or another. We always recommend that if a client doesn't have one, they need to get one. They are good not only for investigations, but also for containment. Thats one area that I think many orgs don't realize the value from.

Scott Nusbaum (TrustedSec): Also don’t just rely on one solution. Have alerts to help detect and address as fast as possible

Justin Vaicaro (TrustedSec): using something like https://github.com/redcanaryco/atomic-red-team. Defense in depth is the key. analyze your security stack, test your logging, and then figure out how you can fill the gaps

Leo Bastidas (TrustedSec): My personal unscientific opinion, MS ATP/Defender, whatever the heck they call it now. You just can't beat the analytics they capture, no one can

X: Hello TrustedSec, how do you bring new hires up to speed? Any tips?

Tyler Hudak (TrustedSec): Throwing them to the wolves on day 1. I kinda kid. When you say up to speed, what do you mean? Are you talking new hires in general or IR new hires?

X: IR specifically

Justin Vaicaro (TrustedSec): strong mentoring, but this also boils down to the individuals willingness to push themselves to learn. read, research, test, write, and present are some of my tools. that i pass down to those i am helping. and then i expect the individual to ask questions..this is huge.

Leo Bastidas (TrustedSec): @Tyler Hudak (TrustedSec) isn't kidding. I was on-site day 2 at TrustedSec. But I wanted to go, beats sitting around the office for inprocessing

X: Any thoughts on bringing in "project manager" types to sit between client and investigator?  I've heard its done by some of the IR firms in the insurance circuit

Tyler Hudak (TrustedSec): This is a great idea, especially for larger incidents. Even internally when no 3rd parties are involved, you should have someone that can intercept requests from the business <--> technical teams

Scott Nusbaum (TrustedSec): Or clients that demand a lot of attention it is great to have someone dedicated to the communication leaving others to focus on the technical

Tyler Hudak (TrustedSec): This allows the tech/IR teams to focus and the business,etc. to still get their info. Plus having a dedicated person in a PM role makes sure all the tasks are getting done

Leo Bastidas (TrustedSec): I love our PMs. They are a godsend IMHO

X: any exploration of how actors (note that I do not mean red teamers) get into networks besides active phishing and active RDP exploration?what about SlilPp, Genesis, UAS, and other markets where cred brokers and IAB (initial access brokers) sell every day and night? If you or your partner/vendor isn't monitoring these then how would you ever know that you were targeted in this way?

Tyler Hudak (TrustedSec): We've def seen activity that likely occurred from credential brokers. Outside of that, highly available exploits are sometimes used, but most often its still RDP and phishing.

Justin Vaicaro (TrustedSec): 99% of orgs are not doing any true counter threat intel monitoring of their orgs imo..when we step into an IR this is identified right off the bat

X: in some ways, though, I see orgs like Intel 471 provide this information for free (assuming you don't already have a contract with them)What's the onus to pay a CTI Analysis and RFI vendor if you could just get the information for free when you need it most?

Justin Vaicaro (TrustedSec): a lot of the average security teams dont even know this. TI is not really a known skill on most security teams imo. at least on the IRs i have worked. and a big misunderstanding of what TI actually is to an organization

X: so we have to build on these capabilities is the answer, and orgs won't know what they don't know (unknown unknowns) until they do

Justin Vaicaro (TrustedSec): most think TI is a threat feed

Interested in more events like this? Join our Slack Workplace today!