OSINT tools make your investigations more efficient but they can also teach you new things along the way. In this post we'll be going over the different things you can learn from open source tools as well as some techniques to make that learning more efficient. Future posts will be deep dives in to different tools using this methodology.
Why learn from open source tools?
A lot of OSINT tools are automating tasks that could be carried out manually. By understanding what these tools are doing "under the hood" you could pick up new techniques or resources you weren't familiar with previously.
The best part about open source tools, especially ones written in Python, is that inspecting them is pretty straight forward. Since they likely exist as plain text scripts (as opposed to compiled code) it can be as simple as opening them up in a text editor or IDE and seeing what they are up to.
What can you learn from open source tools?
By understanding how a particular tool works you could discover:
- Public APIs you weren't aware of - Maybe the tool is leveraging a service you weren't previously familiar with. Now that you know about it, integrating it in to your own tools or workflow could be as simple as reading their documentation.
- Undocumented APIs you weren't aware of - Maybe a given platform has a publicly accessible API that isn't necessarily advertised. Understanding how a tool leverages that API could give you useful insights in to your own investigative methodology.
- Novel techniques - Maybe the tool is doing something unique with several different sources of data. Maybe it's parsing out data in a useful way. Maybe it's allowing pivots you hadn't ever considered.
- Websites or information sources you weren't aware of - Maybe the tool is scraping or querying sites you've never heard of. Knowing these sites exist could send future investigations in a completely different direction.
- Give insight in to how a platform is structured - By understanding how a tool automates a task could give you insight in to the structure of the platform itself. Maybe a tool is extracting data you never knew was accessible on a platform.
- Improve your own tools - Maybe you come across a tool that is really good with async code execution or some other concept you weren't familiar with. Understanding the tools of others could improve your own tooling.
How can you extract meaningful knowledge from OSINT tools?
There a few techniques I use for drawing meaningful insights from open source tools:
- Read their documentation - Seriously. It's likely their README is full of useful information. Sometime a given tool may have a separate "docs" folder where they store things as well. Take the time to look at what the tool creator wanted you to know about the thing they created.
- Check the dependencies of the tool - What libraries or packages is the tool depending to work? It's possible it's leveraging a library specific to a given platform you haven't heard of before. Understanding the dependencies of a tool can also give you insight in to what the tool is actually doing. "aiohttp" implies that lots of web requests could be happening simultaneously. "beautifulsoup" or "selenium" could imply some kind of web scraping. And so on...
- Open the file and look for calls to other sites - This could be as simple as "Ctrl-F" and searching for any mention of "https://". This search alone would tip you off to the use of other sites or APIs to make the tool work.
- Ask AI - Seriously. Drop an individual script in to your favorite LLM and begin asking questions. "What is this script doing?" "What APIs is this script calling out to?" "How is this script parsing data?" If you happen to have Copilot in your IDE (like VS Code) you could try to use that to interrogate the entire project.
Putting it all together
When I find a tool that does something I'm interested in I'll open it up and "Read their docs" -> "Understand dependencies" -> "Look for calls to external sites" -> "Ask AI"
Following that workflow has given me some cool insights in to tools I found a lot of value from. Future posts will feature deep dives in to various utilizing the methodology above. The most amazing part of open source software isn't "getting something for free" but is instead getting insight in to how a person has optimized a particular task.
