Latest / Stage Talks with Bellingcat / "Maintaining An Open Source Toolkit" with Bellingcat Staff and Toolkit Guardians
Transcript
- 0:03You're listening to a stage talk titled Maintaining
- 0:06an Open Source Toolkit. This week we were joined
- 0:09by Bellingcat's Johanna Wild and Bellingcat volunteer
- 0:12community members, Anisa Sabir and Martin Sona.
- 0:16They are all, alongside others, instrumental
- 0:19to the build and maintenance of the Bellingcat
- 0:21Online Open Source Investigations Toolkit, a
- 0:24resource used by over two million people. looking
- 0:27for reliable information on open source tools.
- 0:30They spent the hour discussing how they make
- 0:32decisions on the toolkit and keep the list of
- 0:35applications up to date. You can find links to
- 0:38all the resources mentioned in the talk in the
- 0:40podcast description. This talk was hosted by
- 0:42me, Charlotte Marr on Thursday the 8th of October,
- 0:452026, in the Bell and Cat Discord server. Okay,
- 0:56I reckon we should get going. We have a lot to
- 1:00talk about today. So welcome everyone to this
- 1:03week's State Talk and thanks for joining us.
- 1:06Today we're going behind the scenes of a resource
- 1:08that many people here may be familiar with or
- 1:11could find useful, the Bellingcat online open
- 1:14source investigations toolkit. The toolkit looks
- 1:18at first glance like a collection of useful links
- 1:21and tools, but maintaining a resource like this
- 1:23at scale raises some much bigger questions. The
- 1:27open source investigation landscape is constantly
- 1:29changing. New tools appear, existing ones disappear
- 1:33or change their functionality. Platforms change
- 1:35their policies and technical architecture and
- 1:38techniques that work today may not work tomorrow.
- 1:41Looking at many of the tools that we used to
- 1:43use for Facebook, for example. So there's an
- 1:46interesting editorial and methodological challenge
- 1:49here. How do you decide which tools are worth
- 1:51recommending? How do you test them? And how do
- 1:54you assess their reliability and limitations?
- 1:57How do you provide guidance that is genuinely
- 1:59useful without giving users a false sense of
- 2:02certainty? That's what we're going to explore
- 2:05today, not just the toolkit itself, but the work,
- 2:07judgment and investigation that goes into maintaining
- 2:10a resource used by millions of people. We'll
- 2:14hear from the people behind the toolkit about
- 2:16how they discover and evaluate tools how they
- 2:19deal with change, and what they've learned from
- 2:20maintaining it over the past two years. Let me
- 2:23introduce you to my colleague, Johanna Wild,
- 2:26who oversees the development of the toolkit,
- 2:28and with her are two members of our Bellingcat
- 2:30volunteer community who are also toolkit guardians,
- 2:34Anissa Sabir and Martin Sona. Before they walk
- 2:37us through their work, a short reminder that
- 2:39we will have plenty of time to answer your questions.
- 2:42And you can place your questions in the chat
- 2:44accessible on the right hand corner of your screen.
- 2:47But please remember that this is being audio
- 2:49recorded for the podcast. So if you don't want
- 2:52me to measure your username, please add that
- 2:55as a note in your question. Okay, Johanna, I
- 2:59will leave the stage to you. Thank you. I'm so
- 3:03happy to be here. Hi, everyone. I start by giving
- 3:07you a little bit of background information of
- 3:10what our toolkit is, because I assume not everyone
- 3:14here knows exactly what it is or has used it.
- 3:18Hopefully many, but probably not everyone. And
- 3:21here I start. So the reason why we are here today
- 3:25is because our toolkit just turned two years
- 3:28old, which is really, really amazing. More than
- 3:33two years ago, I had the vague idea that we need
- 3:39a new type of toolkit because Bellingcat had
- 3:42a toolkit before, but it was basically a sheet
- 3:46where we listed tools and it was super helpful.
- 3:50We could see that people use it and come to it
- 3:53quite regularly, but we internally honestly didn't
- 3:57manage to keep it updated. I felt sometimes a
- 4:02little bit ashamed because this sheet was online
- 4:04and we just didn't manage to update it because
- 4:07it takes work. And it still does for this version
- 4:10too, but I knew we needed something else because
- 4:14people use toolkits and they are searching for
- 4:18information about tools. And this is how I had
- 4:22came up with this idea that if we want to build
- 4:27and create a toolkit that makes sense to everyone.
- 4:30We need to do it in a collaborative way and in
- 4:33a way more collaborative way than we had done
- 4:36it before. We had done it mainly within the Bellingcat
- 4:40staff team. So I knew we need to open it up.
- 4:43And this is how we started this. In my point
- 4:46of view, still, I'm very impressed what into
- 4:50what it developed. This toolkit to which many
- 4:54people in our volunteer community contribute.
- 4:57And I just want to talk you through a little
- 5:00bit how do we work on this toolkit, right? How
- 5:04does it work in the background? But let me first
- 5:07of all show to some of you the toolkit and how
- 5:12it looks like. So if you cannot see the toolkit
- 5:15right now, you can imagine it as just a page
- 5:18with different tool categories like geolocation
- 5:22maps and satellites, people, websites, and so
- 5:25on. And in all those categories, you can click
- 5:29on them and you find a list of tools. But what
- 5:33is super important for us is that it's not only
- 5:38a list of tools. So that was super important
- 5:41to me personally, because there are so many great
- 5:46toolkit and other types of resources out there
- 5:50that list tools, and they are really fast. including
- 5:55the newest tools probably on a daily basis, but
- 5:59quite regularly. If you know such toolkits, please
- 6:05share with us. I'm always collecting interesting
- 6:08new toolkits and I like them, but our toolkit
- 6:11is different in that sense. It's not meant to
- 6:14be the fastest. It's not meant to include the
- 6:18newest tools. we do something different. And
- 6:21what we really try to do is we take the time
- 6:24to look at individual tools and to really test
- 6:28them. So if you look at our toolkit, you will
- 6:32see there is always a link. And this link leads
- 6:36to a page. And on this page, we have quite extensive
- 6:41text about the tool. And this is the result of
- 6:44the work of our volunteers who basically test
- 6:47the tools and then they write what they find.
- 6:50And I feel this is the special thing about our
- 6:52specific resource, while there are many other
- 6:55great resources out there, it's really that we
- 6:58test everything in depth and fair the results,
- 7:02right? Because this is what sometimes has been
- 7:06missing in our field. We like to list a lot of
- 7:10tools and we often then already recommend them
- 7:14to others, but sometimes we haven't even yet
- 7:17tested them. And some tools honestly are just
- 7:19not good, or they're a little bit shady, or they're
- 7:22only good for one specific thing, but not for
- 7:26another thing. And that's why we really wanted
- 7:29to look deep into what does a specific tool do,
- 7:34and what does it do specifically for open source
- 7:38investigations, because Some of the tools we
- 7:41use, they have not been built specifically for
- 7:44open source researchers in mind, right? They
- 7:47have been built with other target groups in mind.
- 7:51I always give the example here, for instance,
- 7:55peak visor is a tool that has been built for
- 7:59people who maybe want to go into the mountains.
- 8:02And we now use it for geolocation purposes. And
- 8:06there are many, many other cases like that. And
- 8:09that means that we really have to look at which
- 8:12features of the tool, what exactly of a specific
- 8:15tool can be used and are useful for open source
- 8:19researchers and what other features are maybe
- 8:22not as useful. And yeah, so also what I find
- 8:28is special and it also makes our whole project
- 8:32quite time intense is that we constantly monitor
- 8:37what happens with a specific tool, because tools
- 8:41change all the time. Toolkits get outdated honestly
- 8:45very quickly if you don't look at them constantly
- 8:50and update them constantly. So that's why we
- 8:52really have a very rigorous update rhythm. So
- 8:57every other month, our contributors look at the
- 9:02tools and see what changed and then they changed
- 9:05what they wrote about the tool. So that makes
- 9:08it really, really resource and time intense for
- 9:12all of us, but it also became very clear that
- 9:16this is the real value of our toolkit, right?
- 9:19So it's really very, very useful for researchers
- 9:24to know, okay, I don't have to use this tool
- 9:27because things have already changed. So maybe
- 9:30I can go to another tool that seems more useful.
- 9:35And if we look at the text, the tool reviews,
- 9:41as we often call them, that we have in our toolkit,
- 9:45they have a very specific structure. And this
- 9:48structure really tries to capture the things
- 9:52that are important for open source researchers.
- 9:55They usually start with a description of, yeah,
- 9:59how does this tool work? And tips, how can we
- 10:03use this tool? Where do I need to click? Often
- 10:07things like that. But then we also have sections
- 10:11like, is this tool free to use, right? Because
- 10:15that is super important for many people in the
- 10:18open source researcher community. Not everyone
- 10:21has budgets for tool licenses. Then also how
- 10:26difficult is the tool to use? Because I need
- 10:30to know, do I need 10 hours to learn how to use
- 10:33the tool or can I just go in and use the tool
- 10:37right away? And what we have learned is super
- 10:41important. And that's also one of the sections
- 10:44that we have started to focus on more and more
- 10:48is it's very important to describe the limitations
- 10:51of the tool after testing it because that helps
- 10:55others avoid using a tool that is useless. And
- 11:00equally important are ethical considerations.
- 11:03We really analyze, okay, what type of ethical
- 11:07challenges would come up for someone who uses
- 11:11the tool? And yeah, how could we think about,
- 11:15how could we approach that? And yeah, the structure
- 11:18of our toolkit then also continues. We provide
- 11:22guides and articles if available. We also give
- 11:26information about the tool provider, specifically
- 11:28in which country is the tool provider based.
- 11:32And something that we really also learned over
- 11:35the years and our toolkit has become now two
- 11:39years old, is that it's super important for us
- 11:43to show similar tools for every tool that we
- 11:47list. And if we can, if there are similar tools,
- 11:52we also try to show, okay, and what other similar
- 11:56tools could a researcher also use on top of the
- 12:00tool that they have already on hand. I also want
- 12:08to show you a little bit the background process,
- 12:12because this is something that is usually not
- 12:15so visible. All the things we do in the background.
- 12:19So if you go to a toolkit page, it mainly looks
- 12:22like a page, right? But in the background a lot
- 12:26is going on and I have tried to capture the process
- 12:29shortly how it looks like. So we have a group
- 12:34of volunteers who regularly test these tools
- 12:37and write the tool reviews and once they did
- 12:42that they submit a draft in a system that we
- 12:46have for the toolkit. And once this draft is
- 12:49submitted, Bellingcat staff, at the moment it's
- 12:53in most cases me, we are looking at the draft
- 12:56and might discuss with the person who wrote the
- 13:00draft what we maybe might change, maybe we might
- 13:05want to add some things, maybe not. And once
- 13:07this process has happened, only then we publish
- 13:10it in the toolkit. So there is really a process
- 13:13behind that you cannot see. but it takes a lot
- 13:17of time and that is also super important because
- 13:19we want to make only quality full reviews public
- 13:24of course and then what happens is that every
- 13:28other month the people contributing to the toolkit
- 13:31they check again what is online and check is
- 13:34it still accurate what we have published and
- 13:37this is also where our toolkit guardians come
- 13:40in and two of them are here today they help every
- 13:44contributor every other month with these updates.
- 13:48So in the background, they are supporting not
- 13:51only their own tool reviews, but they are basically
- 13:55supporting the whole team behind the toolkit
- 13:58by helping others to improve, write their tool
- 14:03draft and help them maybe sometimes to test their
- 14:06tools because they have a lot of experience.
- 14:08The process in the background, which I wanted
- 14:10to share with you just because usually I think
- 14:12that is not visible. And before I hand over to
- 14:19our great guardians in a minute, I just wanted
- 14:23to show you why we are still here. So the reason
- 14:28why we put so much effort also at Bellingcat
- 14:31into this toolkit together with our volunteer
- 14:34community is really because it works. So the
- 14:37toolkit went online in September 24, And now
- 14:44we are two years in, and we have had over two
- 14:49million page views in this time, even a bit more,
- 14:53but we lack a little bit of tracking data. And
- 14:57it's very consistent. Many people come back again
- 15:01after they use the toolkit once. So it's slowly
- 15:04going up. More and more people are using it.
- 15:08And maybe just that's always quite interesting.
- 15:12We have really users from all around the world.
- 15:14So of course, from the US, UK, but amongst the
- 15:19top three countries with our most biggest groups
- 15:23of users, it's India, for instance, and then
- 15:26various European countries, but also Brazil.
- 15:30So it's very mixed, which is super, super amazing.
- 15:33And my last insight, maybe for me, it's always
- 15:37super interesting to see what type of tools people
- 15:41like most. So they clearly like to look most
- 15:45at simple tools that you can use very quickly
- 15:49without learning much. And over the two years,
- 15:54very consistently people have always liked most
- 15:58tools that help you find usernames across different
- 16:04social networks or platforms. or a tool that
- 16:10helps you find public telegram content. One tool,
- 16:15TeleGargo, is quite up in the list. And then
- 16:19username tools like Blackbird, these tools have
- 16:23been most successful. But if we only look at
- 16:25our own smaller community, then anything related
- 16:29to geolocation comes up a bit more. Yeah, that
- 16:33was just a short overview. Thank you so much.
- 16:37And yeah, you can also donate to the toolkit,
- 16:41I just mentioned it, because the resources that
- 16:45go in, they can be a lot. So we put a lot of
- 16:49effort in there. And it goes into working together.
- 16:54and in managing the process in the background.
- 16:58So this is what we're really focusing on from
- 17:00the Bellingcat perspective as a team of staff
- 17:05members. But the main work is done by our volunteers
- 17:10and also the two guardians that are there today.
- 17:14Hi everyone and thank you for having me today.
- 17:17It's great to be here. So I oversee the image
- 17:21video category of the toolkit and this is the
- 17:24place that can get you started with the right
- 17:27tools in your visual investigation. And I would
- 17:32broadly divide this tool, this area into three
- 17:36main categories depending on your research needs
- 17:40and they may be different from researcher to
- 17:42researcher. So number one, if there's a video
- 17:45fresh out of a conflict zone and you need to
- 17:48geolocate it, the first thing you would need
- 17:50to do is get the keyframes of the video and then
- 17:55run reverse image searches for that video. So
- 17:58we just have the right tool to get you started
- 18:00with it, which is called like Invid Verification
- 18:03Plugin. And this is how the dashboard of the
- 18:07tool looks like. And from here, then you can
- 18:10run reverse image searches using various search
- 18:13engines. Secondly, if the target of your investigation
- 18:18is a certain person like a bad guy that you want
- 18:21to hold accountable, And you have tried everything
- 18:25to identify who that guy is, and you're having
- 18:29no luck in finding the identity of that person.
- 18:33We have a very powerful category of tools just
- 18:36to get you started on that front, called facial
- 18:39recognition tools. So these are very powerful,
- 18:42but at the same time, these are very shitty tools.
- 18:45So you need to be careful what you upload in
- 18:47the process. And from here, for example, if...
- 18:51The reverse image searches are literally useless
- 18:54when you're trying to identify a person in a
- 18:57video or an image. So for example, if you run
- 19:00a search on PIMIs and it lands you on a page
- 19:03where you can like find more contextual information
- 19:07on the person like their username, email address,
- 19:10then from this category, you can sort of pivot
- 19:13or move to the social media category and people
- 19:16and continue your research from there. And I
- 19:20think One of the most important parts, and I
- 19:25think this kind of shows how this toolkit is
- 19:29evolving. And it also like highlights the roles
- 19:32we guardians and maintainers play is to identify
- 19:36and understand the challenges researchers are
- 19:39facing nowadays. And in this case, it would be
- 19:42the AI generated disinformation out there that
- 19:45has distorted the information landscape. So recently
- 19:49we have published tools on AI DeepFake detection,
- 19:54like DeepFake or METER, and I don't know, it's
- 19:57called HAYA or HAYA DeepFake voice detector,
- 20:00which is very useful for audio deepfakes. So
- 20:04if I can give an example, I use this tool. Like
- 20:10a year ago, there was a claim circulating that
- 20:12the Indian army chief admitted that the country
- 20:15had lost six fighter jets to Pakistan during
- 20:19the conflict in May 2025. And it was kind of
- 20:23a wild claim for an Indian officer to go publicly
- 20:27and say like they lost the fighter jets. And
- 20:31when you hear it for the first time, it does
- 20:33sound like it instantly clicks that this is likely
- 20:37to be false. But then you need to go into the
- 20:40verification process. And this was one of the
- 20:43tools that I used to identify whether his voice
- 20:47was real. And we eventually found out that this
- 20:51was a deep fake. And important to keep in mind
- 20:54is also like when you're using these tools, use
- 20:57other methods also, like reach out to experts,
- 21:00run reverse image searches. And so this was how
- 21:03this particular tool helped me from this category
- 21:06to debunk disinformation. And one of the most,
- 21:11my favorite tools, for example, if you're running
- 21:15reverse image searches and if you have keyframes,
- 21:18you need to organize that data. There's a tool
- 21:21called Pure Ref and we don't have a guide for
- 21:25it yet, but this tool is very useful for organizing
- 21:29your image or reference images that you might
- 21:32be using for geolocation. So I don't know if
- 21:36you can see my screen. This is how the tool looks
- 21:40on the dashboard of the tool. So these are the
- 21:44key frames from a Bellingcat video. And I can
- 21:49sort of arrange the images in any way I want.
- 21:53I could enlarge it, zoom out and rotate it. So
- 21:58playing around with it the way I would want.
- 22:02It's one of the very simplest tools that you
- 22:04can have in your tool kit to organize the images.
- 22:10But still, if you feel like this is not useful,
- 22:12I hope it is, because it's one of my favorite
- 22:16tools. It acts as a board where you can pin your
- 22:19images. We have something more fancy. If you
- 22:24prefer something fancy and sophisticated, we
- 22:27have... image stitching tools to create these
- 22:31beautiful panoramas, just in case if you need
- 22:33to like duplicate an image and you need a perfect
- 22:37picture. And Johanna already highlighted the
- 22:45similar tool section and the limitations. I think
- 22:48it's one of the very important part of this toolkit
- 22:52and effort by the by the maintainers. A lot of
- 22:57the time goes into thoroughly testing the tool
- 23:01and identifying what the limitations are and
- 23:04then ethical considerations which help you whether
- 23:07you should upload an image into a facial recognition
- 23:10tool. And if this convinces you or all of this
- 23:14convinces you that this tool is not the right
- 23:16one, you can always check out the similar tool
- 23:19section for your name. So this is it all from
- 23:23my end. And if you have any questions, please
- 23:27feel free to hear them in the chat and I'll pass
- 23:30over to Martin now. Thank you. Thank you, Anissa.
- 23:36Hi everyone. Thank you for the nice introduction.
- 23:42So over the last three years in the toolkit team,
- 23:47my main area has been social media and the question
- 23:51I keep coming back to as most of us in the toolkit
- 23:56team are, is very simple. Does this tool still
- 23:59work? It sounds like a little bit of a modest
- 24:03question for such an anniversary talk, but it
- 24:05does cover a lot of the job. A tool can have
- 24:10a still working website or an impressive descriptions,
- 24:15while the features you actually need might have
- 24:18stopped working or it works, but only after another
- 24:21login challenge or with another application or
- 24:24with a payment. And helping people understand
- 24:28those conditions has become a big part of maintaining
- 24:32the toolkit. And just as a clarification, we
- 24:37maintain independent guides for these tools,
- 24:39right? So maintaining an entry does not mean
- 24:43we have any sort of association with the tool
- 24:46or its developers. In contrast, that probably
- 24:51means that we don't have any association with
- 24:54the tool or its developers. Our responsibility
- 24:57is solely to the people using the toolkit. So
- 25:01they may be investigating a network or preserving
- 25:05evidence before it disappears. That's especially
- 25:09relevant in my category. And sending them down
- 25:13a dead end just costs time they might not have.
- 25:17The biggest change I've seen over the last two
- 25:22years is that public social media has become
- 25:24hard to research systematically. Just a post
- 25:28being visible on your phone does not mean that
- 25:31you can reliably search for it or collect related
- 25:35posts or repeat the same investigation next month.
- 25:41Especially with the very large online platforms
- 25:44like X and Meta, researchers have face increasing
- 25:49barriers around both technical access and the
- 25:54procedures for getting it if you're eligible.
- 25:58And tools that depend on that access have become
- 26:01a lot harder to maintain or have disappeared.
- 26:05So some disappear, others change the purpose.
- 26:08For example, our OXIE entry now points people
- 26:12towards its successor, OXIEMeNet. So checking
- 26:20whether a familiar name still offers the capability
- 26:23you remember is part of the work. And there's
- 26:28another example specifically with Meta's data
- 26:32research tools. An example was that Meta announced
- 26:37fees for the university hosted environment used
- 26:40to research metadata and Meta's own research
- 26:43environment remained free, which matters. changing
- 26:47environments, moving projects, and understanding
- 26:50how the conditions change, all that takes a lot
- 26:53of time. And sometimes the hurdles are large
- 26:59and intransparent. Excess can become really impractical
- 27:03through the accumulation of these demands, right?
- 27:06So a small newsroom or an independent researcher
- 27:08at a university with a mediocre funding may struggle.
- 27:14long before they reach the actual stage of research.
- 27:18And this is why, this is one of the reasons I
- 27:20want to give the open source tools some attention.
- 27:24So, um, zschreimer, for example, um, collects
- 27:28data. That's, that's a tool that collects, uh,
- 27:32data about supports or supported social media
- 27:35posts that as you browse. So while you browse,
- 27:39it collects the data. You can export that data
- 27:41later or take it into another program for analysis,
- 27:45Forecat, for example. And both are open source
- 27:48projects that are well retained. And that gives
- 27:51researchers a useful way to work with what they
- 27:53can access, even if they don't have a researcher
- 27:56access. We still need to explain the limits,
- 27:59of course, in the toolkit. So a collection of
- 28:03posts loaded in your browser is not a complete
- 28:06picture of a platform. code available also means
- 28:11that people can inspect how your collection works.
- 28:15They can identify mistakes, they can adapt the
- 28:18tool to new questions or help prepare it whenever
- 28:22a platform inevitably changes, right? And for
- 28:27investigative work, being able to explain how
- 28:31you obtain the result really matters. Open code
- 28:37gives us just more opportunity to check the tools
- 28:41that we depend on. It also gives a project a
- 28:44chance to continue if the original developer
- 28:46or the original researcher has moved on to something
- 28:49else. Of course, somebody still has to do that
- 28:52work. An open source software needs maintenance,
- 28:55documentation needs funding. I've seen this in
- 28:59a couple of social media tools and other tools
- 29:03that have. that went through different rounds
- 29:06of funding and have changed hands and teams over
- 29:10the course of the last three years. And we should
- 29:13support the people doing it, including through
- 29:16useful bug reports, contributions, through credit.
- 29:20Of course, OpenCode cannot restore access to
- 29:25data that a platform has closed off. And we need
- 29:29both usable tools and usable access. There have
- 29:35been encouraging developments, too. So we added,
- 29:38for example, Blue Sky Tools in December 2024.
- 29:42We published guides the following March. Blue
- 29:46Sky is an open architecture that has created
- 29:48a lot of room for people to build new ways of
- 29:51exploring social connections. That's really great.
- 29:55Looking through the toolkit's GitHub history,
- 29:58you can see both sides of these two gears. documenting
- 30:05new possibilities and keeping up with methods
- 30:09that have become less reliable. That brings me
- 30:14a little bit to the European rules and what platform
- 30:18transparency means. So I really support the EU's
- 30:21commitment to platform transparency, but I want
- 30:24the rules to be clearer in practice about who
- 30:27qualifies, what they can request. how they can
- 30:32challenge a refusal. So for me, as an academic
- 30:35researcher, it might be really easy to get, not
- 30:39that it might be possible to get access, but
- 30:43it might be a lot harder for an independent researcher,
- 30:47freelancer, or a newsroom. There are different
- 30:49routes for public data and for better access
- 30:51to non -public data with different requirements,
- 30:55like for journalists, freelancers, for civil
- 30:57society researchers as well. Finding the route
- 31:00that actually applies to them should be much
- 31:02easier. A platform should not be able to make
- 31:08a right to research unusable to an exhausting
- 31:13application process. And I've been struggling
- 31:15with this in a couple of my toolkit entries a
- 31:17lot, keeping up with the changes in those application
- 31:21processes. And enforcement also matters here.
- 31:25In October 2025, for example, a commission Preliminary
- 31:29found that Metta and TikTok were failing to provide
- 31:33adequate research access. In December, it find
- 31:36X for transparency breaches that included restricting
- 31:40access to public data for researchers. And of
- 31:43course, those are welcome steps and important
- 31:45steps. And we need consistent follow through.
- 31:49Researchers need ways to resolve access problems
- 31:52while their investigations are still timely.
- 31:57The news cycle is short. The measure of success
- 32:03in general, but also for us at the toolkit is
- 32:07whether people can actually do the research on
- 32:11a legislative level, but also practically for
- 32:13us as a toolkit team. And that's also why, for
- 32:16example, the transatlantic argument concerns
- 32:18below, but the US administration has challenged
- 32:21European rules as unfair treatment of American
- 32:25companies. So what it's actually usually about
- 32:28why these changes happen is money. I want Europe
- 32:31to defend independent scrutiny while making its
- 32:35own rules clear and workable. Researchers on
- 32:38both sides will benefit when they can examine
- 32:44powerful platforms without depending on the platform's
- 32:48goodwill. Because if I've seen anything in the
- 32:50last two years is that a platform's goodwill
- 32:54is something that we cannot depend on. So for
- 32:59me, the toolkit is a practical contribution to
- 33:02that effort. Every guide we review, we test,
- 33:07we write. Every limitation we explain and also
- 33:11broken methods that we flag do help someone else
- 33:15investigate. Every open source project that we
- 33:19help people understand gives that work a better
- 33:22chance of continuing. or gives the newsroom a
- 33:26possibility that they might not have had otherwise
- 33:29because they work out of a country that isn't
- 33:33equipped in the same way as say the New York
- 33:36Times or the New York Post. To finish it up,
- 33:43that's what I've valued over those two years
- 33:46and on the team, like sharing what we learn and
- 33:49helping to keep all this work possible. That's
- 33:53it for me. Thank you so much. Thank you. Thanks
- 33:57first, Johanna, going through what the toolkit
- 34:00means, and then both of the guardians explaining
- 34:04what makes their categories special and also
- 34:07those marks of success. We're going to go into
- 34:10the Q &A now. We've already got a few questions
- 34:13in the chat, but I wanted to start off with this
- 34:15question. Perhaps there's people listening, for
- 34:18example, who perhaps are developing their own
- 34:21tools. or are aware of tools that they find useful.
- 34:27There's now an enormous amount of OSINT tools
- 34:31out there and the toolkit says that inclusion
- 34:33doesn't necessarily constitute an endorsement
- 34:36and that some tools have significant caveats.
- 34:39So what makes a tool worthy of inclusion in the
- 34:42toolkit and how do you evaluate whether a tool
- 34:44is suitable for inclusion in the first place?
- 34:48I'm happy to start talking about that. So this
- 34:52is a very long and ongoing discussion we have
- 34:57in the project because on the one hand, we often
- 35:00want to include everything, but it's just not
- 35:03possible. So we always discuss, okay, should
- 35:07we change our criteria, what we accept and what
- 35:12we don't? So I can tell you how it all started.
- 35:14Since this is a volunteer -driven project, we
- 35:17really started by just asking every volunteer
- 35:20what they are most interested in. So the tools
- 35:23you see in the toolkit are mostly coming from
- 35:27suggestions from the people who also test these
- 35:30tools and had a motivation or an interest to
- 35:34test these tools. But at the moment, we are discussing
- 35:37a lot in the Bellingcat team, those people of
- 35:42us who spend time within the toolkit project.
- 35:45and we might want to change that in the long
- 35:48run and to find a way to let the broader communities
- 35:55we have suggest us tools in a more structured
- 35:59way. We are still figuring out how to do that
- 36:02exactly, so it's not yet happening, but we want
- 36:05in the future that we get signals from the wider
- 36:09open source researcher community what type of
- 36:12tools they would like to see in the toolkit most
- 36:17because we receive a lot of emails and messages
- 36:20from individuals who want really a specific tool
- 36:23to be included. But you know that would just
- 36:26be the opinion of one person. So we are working
- 36:29on a potential system to get the suggestions
- 36:33from more people in and hopefully you will see
- 36:38the system come into place. Maybe not this year,
- 36:42but hopefully next year. And as I wrote in the
- 36:45chat, there is a space, for example, in the Discord
- 36:48server, if you do want to share tools, we have
- 36:51a space for established tools that have been
- 36:55peer reviewed called hashtag tools and sites.
- 36:58And then there's also a space if you are developing
- 37:01your own tool and you're looking for advice,
- 37:04reviews, feedback, that is hashtag. tool developments,
- 37:09which is a really growing channel actually. We've
- 37:12got quite a few people in here working on their
- 37:15own tools, which is really, really cool. FD in
- 37:18the chat says, I used to work as an electrician
- 37:21and lots of people, myself included, become over
- 37:24-focused on tools and neglect skills. I'm new
- 37:27to investigating, but I found myself falling
- 37:30into that mindset again. Does anyone have advice
- 37:33on getting the balance right? Do any of our resident
- 37:37investigators here, I'm talking to you Anita
- 37:39and Martin, want to answer that one? How do you
- 37:42balance the need for skills versus tools? For
- 37:46me, it really helps to have a project. You need
- 37:49something that you want to use the tools on.
- 37:53And I don't think it's wrong to try as many tools
- 37:58as possible. I'm a tool person myself, but really
- 38:02learning appears when you apply your knowledge
- 38:05to something. So that would be my advice is get
- 38:10a project to use those tools and then try it.
- 38:16I don't know if you want to add something Anisa,
- 38:20but otherwise I do and please add to that. I
- 38:23fully agree to what you said, Martin. What I
- 38:27often see is that people are nearly scared of
- 38:31tools when they are beginning to do open source
- 38:34research. I reach out and say, look, please,
- 38:38which tools should I learn first? Give me a list
- 38:40and I learn them all. I would really try to reverse
- 38:45that approach. Very similar to what Martin said,
- 38:48because I feel if you have a clear in your working
- 38:51on and on the way you learn tools, it can take
- 38:56out the fear for some people. If you're a very
- 38:59structured person and you want to go through
- 39:01tools one after the other, that is also fine.
- 39:04But I specifically recommend this approach to
- 39:08attach it to a research case to those who get
- 39:12super fearful by the sheer amount of tools that
- 39:17are out there because it can be super intimidating
- 39:20and I think it doesn't have to be. You don't
- 39:23need to know all the tools. I would say nobody
- 39:26in the project team and the toolkit project team
- 39:29here knows all the tools in the toolkit, right?
- 39:33We are constantly all learning from each other
- 39:36and that is fine. And that's also the strength
- 39:39of it. So let first of all, let the thought go
- 39:44that you need to know all of them. That would
- 39:47be my suggestion. Yeah, I think to add to what
- 39:53Martin and Johanna said, it's important to keep
- 39:57the tools and skills separate. I often get a
- 39:59question like what are the tools that can get
- 40:01a geolocation done when in fact skills are separate
- 40:05from the tools and what works best is to apply
- 40:08them in your research problems and over time
- 40:11you will get the hang of using the right tool
- 40:13relevant to your research. Amazing tips there.
- 40:17I also liked Martin's in the comments. Also keep
- 40:20in mind that your most important tool is probably
- 40:22or always a spreadsheet that is very true. Although
- 40:26our mod Subtleknife disagrees with you and says
- 40:29number one is your brain power. Amazing. I wanted
- 40:35to ask, where do you think the biggest blind
- 40:39spots currently are in OSINT in your respective
- 40:43category topics? So social media and image and
- 40:46video. Are there things researchers are routinely
- 40:50Assuming that they can find actually require
- 40:52much more caution in your two categories. Yeah,
- 40:55where are the biggest blind spots that you see
- 40:57when reviewing these tools? In my category, I
- 41:01haven't come across and you would need to think
- 41:03it out. Maybe Martin has something to add to
- 41:07that, but it's a good question. That's a good
- 41:10question, especially with social media. It's
- 41:15because even when you use a search function,
- 41:19even when you use some kind of tool, you are
- 41:25always being shown what the search function delivers.
- 41:30You never get a full picture. It doesn't matter
- 41:33if you have what kind of access you use. uh,
- 41:41your processing of, of the event, your processing
- 41:44of what, what you read, what you see that leads
- 41:47you to the things that you discover. And that's
- 41:50something that you would be always have to keep
- 41:52in mind that you're only seeing, um, and I'm
- 41:56not saying that this is some sort of conspiracy
- 41:58that they only, that you only see what you want
- 42:00to see, but you have to, you have to really be
- 42:03aware that, um, there is a, there, there is a
- 42:07some, that what you see doesn't, isn't that necessarily
- 42:10a true reflection of what's out there. Maybe
- 42:16I can add, I feel for video, the challenge currently
- 42:21are all these deepfake detection tools that are
- 42:25coming out and we also start including them into
- 42:28toolkit and for a reason we want to include them
- 42:30of course. But often the challenge is that they
- 42:34often give us a percentage probability of how
- 42:37likely it is that a video is a deep fake. And
- 42:42that is not super useful for researchers. What
- 42:47we really need is an explanation why something
- 42:50is a deep fake or not. And many tools are not
- 42:54focused on that yet. So I feel that creates a
- 42:58lot of confusion at the moment. I don't say they
- 43:02are not useful, use them, but be aware that you
- 43:05also need other ways of verification. I feel
- 43:11we all are closely observing how these tools
- 43:14will be changing and hopefully more tools will
- 43:17be focusing on not giving us just a probability
- 43:20score, but a reasoning of how they came to the
- 43:24conclusion and why. Yeah, it's important when
- 43:30you're using deep fake detection tools to find
- 43:34other ways of verifying our deep fake media,
- 43:37like reach out to experts for analysis. If you're
- 43:41using one tool, add two or more, three or more
- 43:45to your toolkit for analysis to double check
- 43:49the results and still look for visual clues in
- 43:53the video, like AI and these artifacts in the
- 43:55video. So that can also be helpful. And at the
- 43:59same time, look for any context information you
- 44:02can find from news platforms, from fact checking
- 44:06organizations who are working on these claims
- 44:09and videos. We've had another question. What
- 44:12has been the most common reason for removing
- 44:16a tool? Broken functionality or a paywall? Has
- 44:20there been kind of common reasons tools have
- 44:23been removed from the toolkit? Oh, yeah, that's
- 44:27a question for me because I typically have to
- 44:31remove them and it's always a difficult decision.
- 44:35So it's not such a difficult decision if the
- 44:38tool just stops functioning. And honestly, the
- 44:41top reason that the often are a bit attached
- 44:46to the tools that we have listed in the toolkit.
- 44:49And we already see for some time that it's getting
- 44:53less useful, something stopped working, but then
- 44:56it's like, oh, well, two things are still working.
- 44:59While five don't work anymore, we still keep
- 45:02it. And then at some point, there is a point
- 45:04where I hopefully remove it, but I feel I often
- 45:09remove. them a bit too late, but that's really
- 45:12the main reason for removing a tool. Other reasons
- 45:19I feel sometimes, since we are still debating
- 45:22internally what type of tools we include, I mentioned
- 45:26earlier that it is mainly based on people's interests
- 45:30so far, but sometimes we then end up having a
- 45:33few tools that are maybe a bit too niche and
- 45:37there might be like five other very similar niche
- 45:42tours out there. So either at some point we would
- 45:45need to include these five others, super niche.
- 45:48And I mean, we can be niche, but I mean really
- 45:51niche. We need to add five others too, or at
- 45:56some point maybe we take this one out. So these
- 45:59are, I would say, the most common common things
- 46:05and sometimes a tool name also just changes,
- 46:09but it's listed under specific name. Then we
- 46:13ask ourselves, okay, everyone knows the tool
- 46:15under the old name. When do we switch to the
- 46:18new one? Because people won't find it for a while.
- 46:21So we keep the old name for a while. So yeah,
- 46:24I don't want to bore you with all the details,
- 46:26but these changes I would say are the most common
- 46:30reasons to take a tool out. Maybe sometimes we
- 46:33find that a tool is shady, that can also happen
- 46:36that a research comes out and then we are debating
- 46:40internally about that. But I think Martin wants
- 46:43to add maybe on this point. Go ahead, Martin.
- 46:46Yeah, exactly. I just wanted to say that because
- 46:49we do have to walk a little bit of a path when
- 46:53selecting the toolkit. So because a lot of the
- 46:56tools that we use and maintain walk on the fine
- 47:00line of the terms of service, which we Sometimes
- 47:05I personally see often as more suggestions than
- 47:10legal binding contracts, but there are some tools
- 47:14that we have. I can think of at least one that
- 47:18I've personally said we need to remove that used
- 47:25a fraudulent payment tech text. It was definitely
- 47:28way over the line in terms of terms of service,
- 47:31but that also used fraudulent payment methods.
- 47:36And so this is generally a line that we need
- 47:38to walk. How invasive are those tools? And in
- 47:44the end, for me, it's personally a question of
- 47:46if I do an investigation and I need to tell people
- 47:51what tool I used, is this a tool that I would
- 47:55be happy to be associated with? And that's my
- 48:01personal line, but of course, it's always a discussion.
- 48:05And this is why regular checkbacks on the tools
- 48:09that you've got in your categories are important,
- 48:12right? How do you keep updated with those updates
- 48:16to tools that you perhaps are happy to have in
- 48:18the toolkit but may have changed slightly? I'm
- 48:21thinking particularly of the social media tools
- 48:23right now. How do you keep on top of that? And
- 48:26if somebody who's listening perhaps notices one
- 48:30of the tools listed in the toolkit has changed,
- 48:33broken, disappeared, or introduced any limitation.
- 48:36What's the best way for people in the community
- 48:38to flag that to you guys as toolkit guardians?
- 48:43Maybe I'll just start with the second question.
- 48:46If you notice that a tool stops working or something
- 48:50else is up with the tool, you can always write
- 48:53an email to toolkit at bellingcat .com. And we
- 48:57react to that, right? If you let us know, look,
- 48:59I tested this tool. It's really not working anymore.
- 49:03And it happens. And we are super grateful in
- 49:05case some of you who listen already did that.
- 49:08It really helps us a lot. We actually remove
- 49:11the tool or adapt information about something
- 49:15not working. And for how to best update it, I
- 49:19hand over. Maybe, Anita, with the video category,
- 49:23a lot is also changing, right? I think both of
- 49:27you are the best to answer that question. In
- 49:30terms of how we stay updated, we update the tools
- 49:35bi -monthly and for that we keep checking the
- 49:39tools when the deadline approaches and we test
- 49:43them regularly in that sense. We visit their
- 49:45websites if the tool provider has an update about
- 49:49their tool. And most of the tools don't change
- 49:53that much, like they stay the same. Yes, there
- 49:56have been times when a certain tool stops working
- 49:59and we have removed those from the toolkit. But
- 50:03yeah, testing them out, like following investigative
- 50:07researchers out there on X and other platforms,
- 50:10if they notice any change, it's a good way of
- 50:13like keeping yourself updated on that print.
- 50:17And another aspect is maybe also that we often
- 50:23try to really look if important parts of the
- 50:30tools change. So for example, what I always do
- 50:33is for command line tools or for open source
- 50:36tools, I keep a close eye on the issue tracker.
- 50:39So when a tool stops working, obviously people
- 50:41will complain at GitHub. That's always important
- 50:45information for me. I always, on the tools that
- 50:48I maintain, I keep a check on if anything changes
- 50:52in the privacy policy, for example, or in the
- 50:56fine print, basically. So whenever the fine print
- 50:58changes, I get an alert. The cookie policy, whatever.
- 51:04And the other thing is, of course, there is news.
- 51:08You can monitor whether a tool comes up in the
- 51:13Google search results. So there's Google Alerts.
- 51:16But yeah, we do not only try to keep up in terms
- 51:21of trying it. Does the tool still work? But also
- 51:24in terms of, is there maybe something that we
- 51:29miss if we just look at the tool itself? Thank
- 51:33you both. We've only got five minutes left, so
- 51:35we've only got time for a couple more questions,
- 51:38one or two. Nick asked, do you think about the
- 51:41toolkit as generating visible transparency and
- 51:44trust in Bellingcat as a broader organization?
- 51:46Perhaps this is one for you, Johanna? I'm always
- 51:49blown away by how transparent you guys always
- 51:51manage to be. It's incredible. Very nice words,
- 51:55Nick. Thank you. Thank you. Yeah, thank you for
- 51:58this question. At least that's the goal, right?
- 52:02I really want that everyone can do this type
- 52:05of research and not only people working at organizations
- 52:10with budgets for tools and maybe the funding
- 52:15to attend courses, right? It's great if you have
- 52:19all these opportunities, but open source research
- 52:23really gives us all the chance to participate
- 52:26and we want to make this possible. And this is
- 52:30also why I invest my time in the toolkit and
- 52:34it takes a lot of time for all of us here to
- 52:37maintain this resource. And as some of you maybe
- 52:41know, maintaining some times it can be a very
- 52:45tedious task in the background that nobody sees
- 52:48and people can take for granted. So I want to
- 52:51thank one more time all the volunteers contributing
- 52:54to this project for two years, because it's not
- 52:57always as visible to anyone what type of effort
- 53:00goes into it. But the goal is really that we
- 53:03want that everyone can do open source investigations.
- 53:07And of course, tools are only a part of it, but
- 53:09they are still an important part of it. And if
- 53:13we help people understand how they can use tools
- 53:16and what type of investigations or specific tool
- 53:20makes sense and for what other type of investigation
- 53:24may be less though, I feel we can Make a positive
- 53:29impact on the field, right? And it's not important
- 53:31whether Bellingcat does the investigation based
- 53:34on these tools or someone else. We really want
- 53:37that important investigations come out and also
- 53:43based on tools and that people know how to use
- 53:47these tools. So this is really our daily motivation.
- 53:52And yeah, we try to be as transparent as possible
- 53:54because It's not about Bellingcat, right? It's
- 53:58not about us as a team. It's about opening up
- 54:02the space that was only available to professional
- 54:05journalists in the past and other researchers
- 54:08with the long training, everyone who can teach
- 54:11this themselves. This is why we are all here.
- 54:15And I hope the toolkit helps with that. It's
- 54:18the open source mindset, isn't it? And as Craig
- 54:22Silverman mentioned last week in his talk, He
- 54:25mentioned that the indicators toolkit search
- 54:29also includes the Bellingcat tools and the Bellingcat
- 54:32toolkit, which is amazing. Shows kind of how
- 54:36we are kind of willing to share the resources
- 54:38that we've collected with other organizations
- 54:40as well. And Craig gave that a shout out in last
- 54:45week's talk, which you can listen back to on
- 54:47our RSS feed or by searching Stage Talks with
- 54:50Bellingcat. Okay, final, final question because
- 54:52we are almost at time. and try to answer this
- 54:56quickly if possible. But I'd love to hear an
- 54:59answer from all three of you. If money was no
- 55:02object, if we suddenly got investment for the
- 55:05toolkit, a large investment for the toolkit,
- 55:08what feature would you want to add to the toolkit
- 55:10that doesn't currently exist? It could be a feature
- 55:13specifically for your category, or it could be
- 55:16a feature for the general toolkit itself. Johanna,
- 55:20do you want to go first? And then we can switch
- 55:23to the other two. This is such a good question.
- 55:26For me, it's a very simple answer, but I think
- 55:29it would have a big impact. I would like to work
- 55:33with way more volunteers, which we could group
- 55:37according to specific tool interests and just
- 55:42spend way more time with them to compare different
- 55:46tools that are similar. Because when we compare
- 55:48tools, we find out so many things. To which extent
- 55:52a tool is shady, what works, what doesn't. If
- 55:55you compare, you find out more. So that would
- 55:58be my dream, way more comparing. And it requires
- 56:01a lot of resources and a lot of effort. Yes,
- 56:05I'm going to jump in because I think that's such
- 56:08a really good answer. I was going to say something
- 56:10really boring, like it would be great if we could
- 56:13just use GitHub to update our entries. But what
- 56:17Johanna said is actually really a lot more important.
- 56:22this is a lot of work and usually when we are
- 56:25maintainer, we're doing the work by ourselves.
- 56:29But having specific groups or having more volunteers
- 56:34who are happy to even maybe also just work with
- 56:38the main maintainer in testing out the tool from
- 56:42their perspective and having maybe a meeting
- 56:47or a brainstorm or whatever in terms of What
- 56:50did you use it for? At the moment, for example,
- 56:53I'm writing a guide for Epiphy, which has a huge
- 56:57wealth of possibilities, and I cannot write about
- 57:01all of them. And it would be really great to
- 57:05have more people come on board and help us. And
- 57:08they don't need to take responsibility for their
- 57:11own tool right away. You don't need to be an
- 57:17expert. Um, if you're interested in tools, if
- 57:19you're interested in making tools available to
- 57:22researchers and the public, um, you're very welcome
- 57:26to come, come join us, uh, whatever your, um,
- 57:29whatever your level of expertise is. As long
- 57:32as you're kind of willing to put in some, some
- 57:35work, of course, but yeah, it would be really
- 57:38great. Thank you. I know that was a really good
- 57:40point. I agree with both Johanna and Martin,
- 57:46basically investing in the long -term development
- 57:48of the toolkit and bringing more people because
- 57:52this is an important project and it should evolve
- 57:56keeping in mind the need of researchers in today's
- 58:01time. Very good answers. I would just have been
- 58:06like, yeah, I want my category to be the biggest,
- 58:10I want big titles. But you all went diplomatic
- 58:12and I liked it. Thank you so, so much for coming
- 58:16in to talk today and thank you so much for your
- 58:18work on the toolkit as well. It is amazing resource,
- 58:22loved by many, used by many, and hopefully used
- 58:25by the people that have attended today as well
- 58:28and listening to this. Please do go check it
- 58:30out. The link to the toolkit is in the chat and
- 58:34it will also be in the description as podcast
- 58:36and we will be back in two weeks time with another
- 58:39stage talk but for now thank you to all of our
- 58:43speakers today and we will see you soon thank
- 58:46you all thank you for listening to the stage
- 58:49talk if you'd like to catch a stage talk live
- 58:52where you can ask the guest questions join the
- 58:55bellingcat discord server by visiting https://discord.gg/bellingcat
- 59:03The music you've heard is titled Dawn by Newer
- 59:07Self and is courtesy of Artlist.