Latest / Grounded Intelligence: Why Most AI Innovations Never Reach the Field / Data Corpus: Why Smallholder AI Starts With What We Don't Collect

Transcript

Data Corpus: Why Smallholder AI Starts With What We Don't Collect · Grounded Intelligence: Why Most AI Innovations Never Reach the Field

0:00
When we talk about large language models, we
0:02
really need data to drive their development and
0:05
delivery of valued services. AI in agriculture
0:08
is only as good as the data it is built on. Data
0:12
exists, but it sits in silos. We don't have a
0:15
data scarcity problem. We have a data fragmentation
0:18
problem. When you are trying to drill down into
0:20
who should make these decisions, how should they
0:23
be made, it's got to be really contextualized
0:25
in the realities in that country or subnational
0:27
context in order to fit the right kind of data
0:31
governance and data sharing. How do we generate
0:34
the local data set so that we... are not actually
0:38
delivering harmful information in the context
0:41
of the environment we're trying to serve. So
0:43
I think the real value from a researcher's perspective
0:47
is more strategic decisions and taking calculated
0:50
risks so that we can improve farmers' livelihood
0:53
and benefit and income at the end of the day
0:56
and the end of the season. Hello, my name is
1:08
David Bergvinson and welcome to Grounded Intelligence.
1:11
This is a series that looks at the responsible
1:12
use of AI by engaging the community on their
1:16
perspectives on the use and application and some
1:20
of the concerns around artificial intelligence
1:22
to empower farmers to realize their full economic
1:25
potential and to manage the challenges of small
1:29
-scale farming in emerging markets. uh we'll
1:33
be going through a whole series on this topic
1:36
starting with the data corpus or data and i'm
1:39
joined by three very insightful guests who are
1:44
dealing with fragmented data systems and transposing
1:48
those into valuable integrated data sets to drive
1:51
models to generate agricultural insights to help
1:54
farmers make better decisions on the ground some
1:57
of the terms you're going to hear are like frontier
1:59
models these are large language models that have
2:02
been developed by large companies like microsoft
2:05
or open ai and you may be familiar with some
2:09
of their names like chat gpt or claude but we're
2:13
also going to talk about small language models
2:14
and the data that is needed to drive those small
2:17
language models which is a critical gap so we're
2:19
going to explore these in the discussion ahead
2:22
so we welcome your suggestions please leave those
2:26
below on questions that have been raised through
2:30
this conversation and suggestions of future conversations
2:33
as we work together towards unlocking the power
2:36
and potential of artificial intelligence to enable
2:39
farmers to realize their full economic potential
2:41
well so welcome everyone today we're going to
2:50
be looking at the topic of data and so in the
2:54
series of grounded intelligence there's many
2:56
aspects of the responsible use of ai And data
2:59
is a core component, because without data, we
3:02
can't really drive the whole system. So this
3:05
is a foundational topic, and we're privileged
3:07
to be joined by three experts, Jawu from IFPRI,
3:14
and Somya and Mike from Athena Infonomics. And
3:20
so I'm going to let each of them introduce themselves,
3:22
just so that you can understand the experts that
3:24
we heard. we have to facilitate this discussion
3:27
so jao would you mind a brief introduction yeah
3:30
sure yeah thanks yeah hi everyone my name is
3:33
jao ku i'm a senior research fellow at international
3:36
food policy research institute based in washington
3:38
dc my background is crop modeling in agricultural
3:42
engineering field so i work on many different
3:46
types of data that goes into the crop model and
3:50
also trying to understand you know what models
3:53
rather generate. So, yeah, it's very interdisciplinary
3:56
research I've been doing at EPRIN for over almost
4:00
two decades now. Thank you, Joe. Mike, over to
4:04
you. Sure. Thank you, David. So, Mike Minkoff,
4:10
I've been overseeing, operating as the director
4:15
of artificial intelligence and digital public
4:17
infrastructure services at Athena. Been particularly
4:24
focused in this kind of ag XAI community of practice
4:28
and working on issues around especially data
4:33
corpus and benchmarking deeply in that community
4:36
of practice for the last year on the heels of
4:38
a 15 year career in global development that span
4:43
working in environment and sustainability, climate
4:46
risk, and more recently kind of data systems,
4:49
data governance and emerging technology. Thank
4:54
you. And Soumya, please. Hi, I'm Soumya Alamuru.
4:59
I am a senior consultant in Athena Infonomics.
5:03
I did a PhD in economics. Now, working closely
5:07
with AP government, we are providing technical
5:10
assistance to Andhra Pradesh, which is a state
5:13
in India, on AI advisory. I'm leading the project
5:17
from Andhra Pradesh. Wonderful. So we have some
5:22
practical field experience. So we're well represented
5:25
here. So our opening topic, as I said, you know,
5:29
around data, when we talk about large language
5:32
models, we really need data to, you know, drive
5:35
their development and delivery of valued services.
5:38
And so let's talk about why data comes first.
5:41
So Jao, would you, you know, give us a little
5:44
bit of a tour around the AI models and tools
5:46
that we'll be exploring? And from your perspective,
5:50
why does the data layer matter so much in realizing
5:53
the benefits of AI in agriculture? Yeah, sure.
5:56
So, yeah, again, because my background is crop
5:59
modeling, I always have that tendency to apply
6:03
all this data topic and similar discussion in
6:06
the context of crop modeling. So bear with me.
6:09
So crop models, for example, it's not exactly
6:13
AI model, but it has the same constraint on data.
6:18
Any model, including AI models, are only as good
6:22
as the data we provide to develop the models
6:27
for. So, for example, if the models are only
6:31
trained or developed based on specific types
6:35
of environments, say, like North American agricultural
6:37
systems and maize farming in large -scale farms,
6:43
it may not have all... a good enough understanding
6:48
of what are the challenges that farmers are facing,
6:51
how to address the risk, and what kind of advice
6:53
it needs to provide under certain climate conditions
6:57
for small -scale farmers in developing countries
7:00
where we are focusing on. So that will introduce
7:04
inherent biases. The models might start giving
7:09
advisories that are not relevant for smallholder
7:12
farmers, not really actionable. It might provide
7:15
advisories to access tools and use chemicals
7:19
that are not even available in smallholder farming
7:22
systems in the countries where farmers are part
7:25
of. So that kind of limitations all start stemming
7:30
from the data. data we provide to the mother
7:34
and the mothers are trained for really need to
7:36
be extensively comprehensive for the environment
7:42
that where our target users are, you know, cultivating
7:48
farms, cultivating crops for. So that really
7:51
worries me. So all these large scale hyperscalers,
7:55
hyperscaling models are using data from English
8:01
language in well -defined farming systems and
8:06
well -documented systems. So it may not capture
8:09
all these nuances or really detailed information
8:13
on smallholder farmer system. So yeah, it might
8:18
miss all these details or these challenges of
8:22
smallholder farmers. Yeah, the context of that
8:24
data is very important, as you pointed out. So
8:27
as we dig deeper into understanding the problem
8:30
of data, I'm going to turn to Mike and Soumya
8:33
on this, because a lot of the current AI efforts
8:38
in emerging markets relies on existing data sets.
8:41
So, Soumya, first to you, drawing on your experience
8:44
in Andhra Pradesh, India, in scaling AI advisory,
8:47
what do you see as the biggest challenges around
8:50
data access to drive AI -enabled advisory services?
8:55
Well, as Jahu correctly said, AI in agriculture
8:58
is only as good as the data it is built on. And
9:03
in regions like Andhra Pradesh, the biggest challenge
9:05
is not lack of AI, but the lack of reliable,
9:09
representative and usable data. The data is fragmented
9:14
across departments. Data exists, but it sits
9:17
in silos. We don't have a data scarcity problem.
9:20
We have a data fragmentation problem. For example,
9:24
agriculture department holds crop data, farmer
9:26
level data, AP planning department manages procurement
9:31
data, AP space application holds satellite imagery,
9:37
and banks hold credit data. None of these were
9:41
designed to talk to each other. Moreover, institutional
9:44
silos lead to duplicated data collection and
9:46
incompatible formats. challenge is last mile
9:52
data gaps. The most granular actionable data,
9:56
what a former example, what a former in one of
10:01
the districts actually planted when they irrigated,
10:05
what inputs they used rarely gets captured digitally.
10:09
Village level agriculture assistants still work
10:12
on paper in many corners of Andhra Pradesh. The
10:19
third, perhaps most critical and deeply structural
10:24
challenge is the mismatch between farmer identity
10:27
and land records. In AB, more than 80 % of cultivable
10:32
land is operated by tenant farmers. Among those,
10:35
a substantial portion is informal tenant farmers.
10:40
They often don't appear in official datasets.
10:47
major challenges that are there in Andhra Pradesh.
10:50
So what you get is AI system trained on skewed
10:54
data if you don't consider these challenges.
10:57
Yeah, good point. Mike, you know, so what is
11:00
the big risk then given what Ramya just said
11:03
around building these systems that are using
11:07
data that really isn't designed to serve small
11:09
-scale producers in mind? Ramya, in your experience,
11:12
what... are the farmer realities that are most
11:15
frequently uh not considered or excluded or distorted
11:19
in the current data ecosystem drawing again on
11:22
your ap work as i said earlier tenant farmers
11:26
they are out of the data structure ap has a significant
11:31
portion of cultivated land it is operated by
11:34
tenant farmers who are invisible to government
11:36
data systems they don't appear in land records
11:39
don't receive crop insurance or schemes uh their
11:44
use of inputs yields are not captured anywhere
11:48
this is one of the uh biggest issues farmers
11:52
are tenant farmers sorry come what percentage
11:56
of the farmers in ap are tenant farmers uh percentages
12:00
wise it is like around 30 to 40 percent but 80
12:04
percent of cultivable land is operated by tenant
12:07
person tenant farmers so the but those farms
12:14
would have data though it's just that they don't
12:16
have data on the tenant farmer who's cultivating
12:19
that one yeah the the like the kind of crops
12:22
they saw or the inputs they use or the market
12:26
or the yields they they arrive at are not captured
12:31
anywhere and most of these farmers hold most
12:35
of these farmers would be cultivating in less
12:38
than two acres of land And these government data
12:43
collection systems have thresholds like less
12:49
than one acre or two acres are not captured in
12:53
the data systems. So that's a big challenge.
12:56
Yeah, it is. Second is land is disproportionately
13:02
registered in men's name. In Andhra Pradesh or
13:07
in India, farming is a family. It's never one
13:12
person's job. The whole family, husband and wife
13:16
and their children gets involved in the farming.
13:19
In most of the cases, probably women take farming
13:24
decisions, but the lands are not their names.
13:27
We don't consider women farmers as farmers and
13:31
the data doesn't capture that part of it. Yeah,
13:35
you've raised some two very good examples around
13:38
inclusion. And we'll circle back to those later.
13:42
You know, one of our other big challenges, as
13:44
you said, Soumya, is the fragmented data. And
13:47
so, you know, going to this topic of shared data,
13:51
we call it a corpus or a body of data. Jau, I'd
13:56
like to get your perspectives on the discussion
14:00
papers that AgXAI is working on, especially as
14:03
we talk about data corpus and the sharing of
14:06
data. Why is this concept so important in addressing
14:11
this isolated data ecosystem that agriculture
14:14
faces? Right. So there are two things for this
14:19
question. So one is to really understand what
14:22
kind of language the farmers are speaking, even
14:24
in just agriculture in general, which may or
14:27
may not be similar to a general purpose. like
14:31
large language models are trained on. Agricultural
14:34
languages and norms and specific terminology,
14:38
some are really colloquial terms that only kind
14:41
of used in, you know, farming context, again,
14:45
like may or may not be so central to the large
14:47
language model. So the providing and managing
14:51
and consolidating this agriculture data corpus
14:54
is really important if we are serious about providing
14:58
really actionable and, you know, meaningful advisory
15:02
for farmers. They wouldn't take... advisory seriously
15:06
if we use very generalized language not really
15:10
speaking the same tone or same kind of the terminology.
15:15
So I think that's really important to communicate
15:17
better with farmers and understand better what
15:20
kind of language they are speaking. Also in generally
15:23
this kind of corpus sharing shared data corpus
15:26
provides again the more broader context to the
15:31
types of information and other values that we
15:34
are providing. It's not just agriculture on the
15:38
ground and field level expertise and field level
15:41
practices. It also needs to touch on labor, agricultural
15:46
labor and financial access and everything in
15:48
rural communities also needs to be captured.
15:51
And again, this just cannot be done, cannot be
15:53
captured in one single or limited number of data
15:56
sources. So this shared data corpus is really
15:58
important for for reaching out to farmers and
16:02
providing meaningful advisories. Yeah, so you
16:04
lead to an interesting point here, really around
16:07
the balance between scale and specificity of
16:10
these data sets. Sami has already mentioned that
16:12
large segments of society are missing in these
16:16
representative data sets. So how do we find the
16:20
right balance? Because going into that very granular
16:23
data, that costs quite a bit of money to collect.
16:27
So how do we find this balance between the specificity
16:30
and scaling these geographically large data sets,
16:34
at least from the CG perspective. Yeah, I know.
16:38
So this is really challenging in a way. As a
16:41
research institution, things are rapidly changing
16:44
so fast. So, yeah, it's even just hard to just
16:48
keep. keep track of what's going on, where we
16:53
should focus. So getting the right balance is
16:56
absolutely critical. I think the general agricultural
16:59
knowledge and to some extent, the country level,
17:03
large scale, large economy level information
17:05
and data and knowledge are very well captured
17:08
in hyperscalers and global scale corpus kind
17:13
of exercise already. So I think what we can do.
17:16
again, as an institution like CGIR, is to provide
17:20
more detailed, focused, really thoughtful capturing
17:25
of the processes that farmers are, you know,
17:28
challenges that farmers are dealing with. Like,
17:32
for example, if we are to provide advisories
17:36
for, like, say, chili farmers in India, the general
17:40
farming practices or the best practice for chili
17:42
farming in India. must be already captured very
17:46
well in the larger corpus or the existing corpus
17:49
already out there. But what we can do is, what
17:51
is the latest trend of the managing charities?
17:55
What is the market demand, emerging kind of consumer
17:59
demand, and something more actionable, more dynamic?
18:05
types of data, I think that will be probably
18:07
more meaningful effort that we can provide so
18:10
that we can be complementary and to reach the
18:13
balance of the specificity and scaling. So that's
18:17
a great segue into some broader issues around
18:19
governance incentives and trust, which is so
18:22
important. If you betray the trust of the consumer
18:25
in this, it'll take a long time for them to come
18:28
back to this technology. So Sami, I'd like to
18:30
ask you around data governance. it's a term that
18:34
feels abstract so why don't you give it some
18:37
context as it relates to the reality on the ground
18:40
in ap around data sharing access and responsible
18:43
use of personal information so i can talk about
18:47
um ap ai advisory is still growing in in the
18:54
sense like we really don't have advisory systems
18:57
that uh actually considers granular level data
19:01
or use a data corpus which is which can which
19:07
can includes granular level to aggregate data
19:10
but what a representative data would be looking
19:16
like is to cover all agro economic zones in ap
19:22
and major eight to ten cropping systems and irrigation
19:28
systems like rainfed, canal or borewell or tank
19:32
irrigated context and full spectrum of farm size
19:36
from 0 .5 acres to 20 plus acres. If a data set
19:42
covers all of these aspects, we can say that
19:45
is a representative data. And if I have to give
19:50
a recent example of a good AI model from Andhra
19:55
Pradesh, A model that uses Godavari Delta region,
20:02
it's a real -time Godavari, Godavari is a river
20:05
in Andhra Pradesh. It is real -time Godavari
20:07
flood advisory is a game changer. When the state
20:11
integrated IMD weather feeds with canal flow
20:14
data and farmer location data, it became possible
20:18
to push geo -targeted text -based flood warnings
20:21
to farmers in specific regions. These flood alerts
20:24
help farmers in many ways, not just the safety
20:27
point, but to make a decision on sowing or harvesting.
20:35
And recently, like last year, like a few, another
20:38
example I can give is ahead of Cyclone Month
20:43
into 2025. AP Ames, which is a data repository
20:50
system that Recently, the Andhra Pradesh government
20:54
has started building and delivered 3D flood simulations
20:58
and early warnings via SMS and IVR to millions
21:02
of farmers, which helped in protecting 800 ,000
21:06
hectares from flood damage. So, I mean, that's
21:14
really important, especially against the backdrop
21:16
of climate change and extreme rainfall events.
21:21
i i i turn to you mike around okay at a higher
21:26
level who's making the decisions and on what
21:28
data is made publicly available and how is it
21:32
regulated maybe just drawing on your experience
21:34
again with ap or perhaps african nations that
21:38
you've worked with Yeah, I mean, I think that
21:42
the question of who is making those decisions
21:45
or who should make those decisions is an interesting
21:48
one and a challenging one. And I think what we've
21:51
seen, and I know, you know, in other fora, Jiawu,
21:56
we've kind of touched on this, right, where there's
22:00
definitely not one size fits all. And, you know,
22:04
the models that are successful, I mean, even
22:06
if you look at India, India's got a variety of
22:09
models in the way that they are approaching kind
22:12
of the challenge around building data corpus
22:14
or corpora. There's a kind of federated structure
22:19
that's been built at the federal level through
22:21
Open Agronet. AgriStack and Vista, you know,
22:26
now Bharat Vistar. And then there's kind of state
22:29
level deployments or adjacent deployments. Andhra
22:32
Pradesh is kind of an adjacent deployment to
22:35
that. Maharashtra has a state level. And all
22:38
of these have kind of different functions, you
22:40
know, different features and different roles
22:42
of government. And then if you look at, you know,
22:45
context in certain countries in sub -Saharan
22:47
Africa. It's very different as far as what is
22:50
the existing digital infrastructure capacity
22:52
and capability? What is the existing government
22:54
kind of capacity and levels of trust in government
22:57
to fund and monitor and manage something like
23:00
this effectively? So that's all a kind of long
23:05
preface to say, I think when you are trying to
23:09
drill down into who should make these decisions,
23:11
how should they be made? It really has to be.
23:16
contextualized to the political economic reality,
23:18
the techno -institutional realities, the sociocultural
23:23
realities, choose your hyphenate terminology,
23:28
but it's got to be really contextualized to the
23:30
realities in that country or subnational context
23:33
in order to fit the right kind of data governance
23:38
and data sharing processes and protocols. uh
23:43
that that can really kind of sustain um and and
23:47
those need to be designed with real clear -eyed
23:51
on kind of the what's in it for them for all
23:54
of the parties involved right and like how how
23:57
are they gonna you know what are the meaningful
23:59
incentives to sustain contribution and collaboration
24:02
through this data sharing mechanism and partnership
24:04
and so whether that's a government entity that's
24:08
contributing whether it's a research institute
24:11
or like network like cgi are whether it's private
24:14
sector actors that you that have great data but
24:16
have competitive interests and maybe not sharing
24:18
all of it uh you know all all of them have their
24:22
different sure realities yeah well let's let's
24:26
pivot to joe then from the research context perspective
24:29
around the data sharing uh and the incentives
24:31
for them joe are there you know incentives or
24:35
policies in place that facilitate data sharing
24:38
across the cg and make it accessible to a broad
24:41
range of stakeholders in country absolutely so
24:44
cgir As the largest publicly funded agriculture
24:48
research institution in the world, we do have
24:51
a policy since 2021 on fair and open data access
24:56
across all the CGR centers, including my center,
25:00
IPRI. So all the research output, research data,
25:03
research publications should be in the public
25:05
domain. And eventually, we are given a little
25:09
bit of embargo period while we are developing
25:12
and finalizing publication. But the idea is that
25:15
everything publicly funded project that produced
25:19
should be in the public domain. But having said
25:21
that, we do have a challenge. We have a little
25:24
bit of issue or say mismatch between the purpose
25:27
of this policy and what's really useful purposeful.
25:31
like say training AI and LLMs and AI models for
25:35
artificial services. The nature of our research
25:38
is to develop and generate knowledge around how
25:43
we improve farming and how we improve productivity,
25:47
etc. All this towards good outcomes and better
25:51
outcomes in food systems. This may or may not
25:55
be the reality in the farmer's field. So we often
25:59
have, like say, agronomic trials, very controlled
26:03
environment where we know what's going in and
26:06
what's coming out. We can measure perfectly.
26:08
We can observe perfectly so that we have better
26:12
understanding of what works and what doesn't.
26:15
So again, this may or may not be really relevant
26:18
for smallholder farming systems. So we always
26:22
have a little bit of the issue. or trying to
26:27
fully understand and digest the research output,
26:31
research findings, and turn that into agronomic
26:33
or farming advisory is always a little bit of
26:37
kind of, there is a human really need to be in
26:41
the loop so that interpret that research finding
26:44
into the right context for farmers. So that's
26:46
a little bit of issue. So having said that, research
26:49
data is, again, still fully open. We have a policy.
26:53
We also have a governance framework we are developing
26:56
that defines four specific roles in this data
27:00
ecosystem. Data owner, data lead, data producer,
27:04
and data users. And we are also articulating
27:07
whose responsibility, at what stage, at what
27:10
level. So who will make such decisions on data
27:14
sharing and opening data. yeah in different different
27:19
ways so yeah it's a lot of very complex issues
27:22
and we are dealing with again a research institution
27:24
can do so much so we really need partners on
27:27
the ground to make sure what we are generating
27:30
is a fully useful and meaningful for farmers
27:33
too and all this is happening in the context
27:35
of things moving so fast in the ai world you
27:38
mentioned frontier models and for for those not
27:41
familiar these are you know the models that have
27:43
been generated uh by anthropic called claude
27:47
or through open ai chat gpt gemini in case of
27:52
google meta etc and they're moving at such a
27:57
pace and yet the clients that we're trying to
27:59
serve namely smallholder farmers and emerging
28:01
markets that's a very different ecosystems and
28:05
so there's some hard choices and trade -offs
28:07
to to manage here So in light of that comment
28:13
and these rapidly moving frontier models, how
28:18
do we generate the local data set so that we
28:21
are not actually delivering harmful information
28:25
that isn't in the context of the environment
28:27
we're trying to serve because it's coming from
28:29
an advanced economy data set? How do we sort
28:33
of round that off? And Mike, maybe just give
28:35
some of your high -level perspectives on this,
28:37
given your India experience. Yeah, I mean, what's
28:43
really one of the interesting parts of this,
28:46
right? A, there's kind of a juxtaposition or
28:49
tradeoff between the use of the frontier models
28:51
that you described and do they have the good
28:55
data in them, period. But even if they do, there's
28:58
a different question of... Do we want to be using
29:02
frontier models in these contexts all the time?
29:04
Or, I mean, increasingly we're seeing how powerful
29:07
and how accurate small language models, localized
29:11
small language models can be if they have the
29:14
right data in hand. And there's also these kind
29:17
of ongoing subtext questions around... sovereignty
29:21
of models and, you know, on -premise solutions
29:25
and on -site data storage and management. And
29:29
so I think there's a couple layers to it. And
29:32
I think, you know, to collect that data, right,
29:35
as small language models get more powerful. you
29:40
actually, it starts lowering the burden of how
29:43
much good data you need to get to be effective,
29:47
to be able to develop effective localized models
29:50
that can... fill in some of these gaps that these
29:54
frontier models don't have. And that also then
29:57
helps you solve for these sovereignty challenge
29:59
and on premise data management challenges and
30:02
compute cost and infrastructure cost challenges
30:05
that are, that are real, real limiting factors
30:08
in a lot of low and middle income country context.
30:10
So, so I would say, you know, Continuing through
30:16
programs like what we're doing in Andhra Pradesh
30:20
right now or other programs, that gives you really
30:22
an opportunity to start collecting this localized
30:25
data or work that CGIR can do and different approaches
30:30
to complementary data collection. If I was capturing
30:35
what you were saying correctly, those present
30:40
opportunities to start building these complementary...
30:43
more localized data sets that can feed into whether
30:47
it becomes a frontier model at a large language
30:50
model scale or it becomes small language model
30:52
deployments. Either way, I think, and from what
30:57
I'm seeing, it seems like small language models
31:00
will increasingly be the way to go. I think that's
31:03
kind of where the opportunity continues to emerge.
31:07
Well, especially around the data sovereignty
31:09
and model sovereignty that you mentioned, that's
31:11
important. you know i and you brought this up
31:15
earlier on jawu is you know we're using that
31:17
data to make decisions through modeling and so
31:21
you know you also pointed out that we need quality
31:24
data to drive a quality model that delivers relevant
31:27
results so what does good data look like to you
31:32
that enables farmers advisories or system level
31:36
interventions to be effective using this new
31:40
technology Right. So through our partnership
31:44
with Digital Green, who are already scaling this
31:49
agricultural advisory, the AI -powered agricultural
31:53
advisory to many, many farmers. So we actually
31:56
had a chance to look into what kind of questions
31:59
farmers ask. And there are just two groups of,
32:02
you know, two. I guess a bucket of questions.
32:05
One are very strategic, one are very tactical.
32:08
So tactical questions like what should I do right
32:11
now for addressing my problem? What should I
32:14
do today or tomorrow? And then there are more
32:17
strategic questions. Okay, how do I change my
32:22
farming to be more profitable, more resilient
32:24
to climate change and things like that? What
32:27
we saw was that AI models are currently used
32:30
for very tactical questions only. And sometimes
32:32
farmers even ask questions they already have
32:34
no answer to. They are just testing how good
32:37
this model is. So I think the real value from
32:41
a researcher's perspective is more strategic
32:44
decisions and taking calculated risks so that
32:49
we can improve farmers' livelihood and benefit
32:51
and income at the end of the day and the end
32:54
of the season. the more sophisticated system
32:58
we are developing and more comprehensive data
33:00
we are providing, I think we can enable a farmer
33:03
to think more medium to long -term kind of goals
33:08
and developing vision and plan for that so that
33:11
they can really improve their livelihood over
33:14
longer term and more thriving in farming. Yeah.
33:18
And I think, you know, in my experience, the
33:21
market integration is a really important component
33:24
of this so it's the advisory to produce the crop
33:26
but as you point out thinking even beyond that
33:28
strategically what market is that crop going
33:31
to go into and planning accordingly so very good
33:34
point sami i turn to you around examples where
33:38
data design and collection has materially changed
33:44
the outcome of a project so you started with
33:47
a protocol for either pulling in existing data
33:50
or even surveying for new data but found that
33:52
they weren't meeting the needs or delivering
33:56
the kind of benefits you anticipated. So how
34:00
has that played out in advisory services in AP
34:04
where you've seen this kind of situation where
34:06
data quality or access has really limited the
34:09
ability to deliver the impact desired by the
34:13
government, the key stakeholders and farmers
34:17
themselves? Yeah, as I mentioned earlier, like
34:21
I just want to add one point to the when we are
34:26
talking about what kind of data do we have to
34:28
collect and how we how do we on what basis we
34:34
build AI systems, the data collection should
34:38
take a bottom up approach. Usually what happens
34:42
is like we have district level state level data
34:45
and then we just. use averages to proportionally
34:50
distribute to lower levels village level or farmer
34:53
level but if we start from the farmer level or
34:56
village level that what data exists and then
34:58
add up to you know a mental level or district
35:02
level and or state level your data would be much
35:07
more close to the ground and it would be reliable
35:10
so recently in Andhra Pradesh some real efforts
35:17
have been put in place to make this data corpus
35:23
the government is working on building large integrated
35:26
data repositories like ap aims which is andhra
35:30
pradesh agriculture information management system
35:32
that aims to bring various data points including
35:36
cropping patterns soil health pest management
35:39
weather data all into a single platform So using
35:44
this platform recently, they have started delivering
35:48
weather alerts. It could be like cyclone alerts
35:53
or whirlwind alerts. These are really having
35:58
an impact on farmer life. These have a positive
36:04
impact on their farming decisions, also on their
36:08
safety nets. that is one example i could give
36:13
in and then if i have to talk about poor quality
36:19
ai advice which are like there are many ai agri
36:23
fintech companies are releasing a lot of ai advisories
36:28
which depend on standard university or you know
36:32
textbook guidance which are like useless to particularly
36:38
regional smallholder or tribal area farmers.
36:46
The advisory has to be totally related on ground
36:52
-rooting or close to reality data. I'm happy
36:59
to say that in Andhra Pradesh, these efforts
37:02
have been initiated and they started giving good
37:06
results yeah so and and i guess you're mapping
37:12
these data gaps in the case of under pradesh
37:15
and addressing them and making sure they're relevant
37:17
which is really encouraging to see especially
37:19
when you point out the issue of um you know farmers
37:24
that don't hold title or women who are often
37:26
out of the loop when it comes to formal extension
37:30
services and so this really opens up a huge opportunity
37:33
for inclusion if it's done properly so um that
37:37
leads to my could i just add one little bit on
37:40
this on the andhra pradesh sure yeah just very
37:44
quickly i mean one thing that's very cool about
37:46
the what the government of andhra pradesh is
37:50
aiming to do and some of the work that we're
37:52
doing it extends beyond you know agricultural
37:55
data as we're talking about it and includes and
37:58
a big focus of of the program uh we're involved
38:01
in is looking at kind of the horticultural data
38:04
the livestock data the the fisheries on aquaculture
38:07
data as well and it the the vision is to bring
38:10
it all into an integrated space where all of
38:13
this data can kind of talk to to each other in
38:15
addition to kind of market and, and kind of financial
38:18
access information and data and climate and weather
38:22
data. So you can get this really, you know, kind
38:24
of job. Like you were talking about this, this
38:26
more sophisticated strategic inferential advisory.
38:31
And, and so that's kind of the direction it's
38:33
heading. And right now there's a lot of data
38:35
ecosystem gap mapping and, and whole filling
38:38
and, and kind of integration that the team is
38:42
doing. But I, I just, I think that's an important
38:45
kind of dimension to highlight as well as the
38:48
possibilities that are there. That's leading
38:49
into our last question. Thanks, Mike, around
38:52
our last two questions around looking forward.
38:54
So what do you see in the future? I'm going to
38:56
ask you to frame both questions and we'll do
38:59
a roundtable on your responses. The first one
39:01
is, if we get the data corpus right, it's federated,
39:05
responsibly assembled, contextually relevant
39:08
data set. What becomes possible for AI and agriculture
39:11
that isn't possible today? and the second question
39:15
is what worries you most if data remains fragmented
39:20
so those two questions as we close off on this
39:24
topic of data corpus so let's turn to jao first
39:27
yeah sure so um it's not totally correct to say
39:34
it's not possible today because something's already
39:36
happening but it's hyper personalized assistant
39:41
almost like persistently with you to make suggestions,
39:45
to make recommendations just for you. I think
39:48
that will be really, you know... transformative
39:52
to some extent for smallholder farmers who understand
39:56
what kind of resources you have, what kind of
39:58
challenges you have, what's the best course of
40:00
action, the minimal risk, et cetera. Right now,
40:04
I think when you start chatting session with
40:06
any chatbot, it kind of starts from scratch or
40:09
have only very limited session information, only
40:13
focus on what you provided for the chatbot. But
40:17
if all this data... sources are connected as
40:21
a more digital public infrastructure. It can
40:24
pull many different types of data from different
40:26
sources. So it can provide more customized and
40:30
really personalized advice. Personalized advisor
40:33
for farmers. Yeah, exactly. And I think it can
40:38
eventually lead to something more physical. I
40:41
think there are a lot of innovations happening
40:44
in robotics. Again, But in a way, LLM, yeah,
40:49
so we have been hearing LLM is going beyond language,
40:54
already past the language barrier. It's just
40:56
becoming multimodal, becoming operating system
41:00
for robotics as well. So I think you can really
41:03
pave ways for more physical and different types
41:08
of AI transformation that we can lead to. Well,
41:11
I'm already seeing, like, you know, drone integration
41:13
of AI and analytics, for example, today. It's
41:17
a hint of what's to come. Mike, over to you,
41:20
your, your thoughts on those two questions. Yeah.
41:24
I mean, I largely agree with Joe Wu as far as
41:27
like, if we get it right, I think it presents
41:29
this opportunity, you know, like at this point.
41:32
I've integrated frontier model tools like Cloud
41:36
or ChatGPT into my daily work stream because
41:39
it's a super useful personalized advisor on the
41:42
types of things that I use it for. It has this
41:46
global library of knowledge that it can infer
41:50
from and then that gets complemented by the personalized
41:53
exchanges that I have with it. If you get that
41:56
at the farm level for these And it's something
42:00
we haven't explicitly stated, I don't think,
42:01
in this conversation, but is generally well known.
42:04
There's a huge extension gap for the smallholder
42:09
and tenant farmers. You're talking 1 to 6 ,000,
42:12
1 to 10 ,000 in some places. one extension officer
42:17
for 10 ,000 small -scale producers or tenant
42:20
farmers. And so a lot of these farmers don't
42:24
necessarily have the advisory access, generally
42:27
speaking. And then if suddenly they can have
42:28
a personalized assistant that can tell them not
42:31
just, like you said earlier, David, how to plant
42:33
rice more effectively, but actually why they
42:37
should be focusing more on their livestock than...
42:40
you know, farm crops this season because of market
42:43
conditions, anticipated weather, and who knows,
42:45
geopolitical realities and global, you know,
42:48
whatever factors become integrated into this,
42:51
that becomes a super powerful and potentially
42:54
very meaningful livelihood resource. And another
42:56
thing we haven't quite said in this is, right,
42:59
agriculture operates in seasons, right? We're
43:02
not talking like a week or a month. We're talking
43:05
about... six months, three months, a year of
43:08
someone living off in subsistence or near subsistence
43:11
conditions being derailed, that's a huge vulnerability
43:16
and kind of moral hazard risk. So there's real
43:20
powerful potential and excitement there. On the
43:23
worry side, building these systems is very hard,
43:28
especially in the context we're talking about
43:31
with the level of... integration, that kind of,
43:35
you know, multi -state, multinational, you know,
43:39
global levels that gets sustained, especially
43:42
under kind of very dynamic geopolitical times
43:46
and circumstances. So, you know, I worry about
43:50
if the data remains fragmented and biased, then
43:56
we start putting out digital tools. which agriculture
43:59
has done many times in the past. And farmers
44:02
are like, this doesn't actually help me. And
44:04
then it just becomes a lot of investment that
44:06
doesn't lead to... And you betray the trust of
44:08
the farmer. And you betray the trust of the farmers
44:10
yet again with your newest, latest, and greatest
44:13
digital attempt, attempted digital intervention.
44:15
So, yeah. Good points. And Sami, final word,
44:19
boots on the ground perspective. Love to hear
44:23
your perspective on these two questions. i totally
44:26
agree with jahu and mike uh if we get the data
44:29
corpus right we can give truly personalized advisories
44:34
not district level not village level recommendations
44:37
but plot level advice that accounts for your
44:40
soil type your variety your irrigation access
44:44
your historical pest patterns and your market
44:47
access point i want to give one example that
44:51
we did recently in andhra pradesh we are trying
44:56
to get plot level granule granular data in in
45:02
one of the regions which is a banana rich region
45:06
most of the farmers are have banana plantations
45:12
we try to collect the data on which farmers are
45:17
using fruit care activities like bunch covering
45:20
or injection giving injections for to get it
45:24
uh fruit quality to our surprise we came to know
45:27
that out of 600 farmers there are only 12 farmers
45:31
who are following these measures rest of the
45:34
farmers don't even know are they're not aware
45:37
of or they don't know how to do it now the if
45:42
if we get this data that out of 600 12 farmers
45:46
are only following the fruit care activity that
45:49
granular data can truly target those rest of
45:55
the farmers who are not following fruit care
45:57
activities. And then the guidance can go to them
46:00
how to improve the fruit quality and get better
46:03
price in the market. So the real data corpus,
46:09
like real ground level data corpus can give such
46:15
advisory. And second point on tenant farmers
46:18
and women farmers or tribal farmers who have
46:20
been excluded from land records can be visible,
46:25
may get visible through alternative data like
46:29
input purchase records or market transaction
46:32
history. So we can bring them into system by
46:36
covering these data sources. Third point I want
46:40
to... uh talk from the government perspective
46:43
governments can have real -time granular data
46:47
on what farmers are actually doing not what they
46:51
are supposed to be doing policy can be adjusted
46:53
mid -season rather than waiting for annual assessments
46:56
estimating losses and gains and then taking a
47:00
decision what worries is If the data models or
47:09
AI systems are not built on representative data,
47:15
AI will deliver high -quality personalized advice
47:19
to large connected progressive farmers only and
47:24
generate potentially harmful advice to smart
47:26
holders. That widens productivity gap. That widens
47:32
wealth gap. it will increase the inequalities
47:36
which are already existing in the agriculture
47:39
field. AI becomes another mechanism for agricultural
47:44
polarization. Yeah, we don't want to create a
47:49
digital divide 2 .0. So, you know, responsible
47:52
collection, use and delivery of data insights
47:55
to farmers, policymakers and value chain actors
47:58
is critically important to unlock the value of
48:01
this. Well, thank you to all three of you for
48:04
a very insightful conversation. Sonamia, thank
48:06
you for your boots on the ground perspective
48:08
from Andhra Pradesh. Mike, from your broader
48:10
perspective of translating this into projects,
48:13
policies, and impact. And Jau for wrangling data
48:19
across a wide range of international organizations
48:22
and delivering value to national stakeholders
48:24
through data models and a wide range of tools.
48:28
Thanks to all of you. You've raised some actually
48:30
interesting questions for a follow -up conversation,
48:33
especially around personal identification information
48:35
and other topics that we'll follow up in Grounded
48:39
Intelligence. So thank you all and a very exciting
48:44
journey that we're on. And thank you for your
48:46
contributions towards making it impactful and
48:49
real for smallholder farmers around the world.
48:56
Today, we're going to be talking about apples
48:58
and not what you think. We're going to be talking
49:01
about the diversity that exists in an orchard
49:04
of apples and that no two apples in that orchard
49:08
are exactly the same. That variation depends
49:11
on the variety, depends on the micro environment
49:15
in which that apple was produced, depends on
49:18
the soil, the level of water, how it was cultivated
49:22
or pruned by the farmer. So when we talk about
49:26
variation not just in apples but variation in
49:29
our data systems to support modern agriculture
49:32
we're talking about data that contextually is
49:35
very different and we need to make sure that
49:37
that variation is captured in the data that's
49:40
going into training models especially models
49:43
to support advisory services for farmers so that
49:46
means making sure that we capture the diversity
49:49
of production systems Farmer profiles, especially
49:53
women that are often underrepresented in these
49:55
data assets for a variety of reasons. That we
49:58
capture variation in the soil type in which these
50:03
crops are grown. And also making sure that that
50:06
data that is being captured has associated with
50:09
it what we call metadata, which is sort of smaller
50:11
bits of data that actually describe the context
50:14
of how that data was collected. the parameters
50:17
that would help you understand how to best use
50:20
that data in order to deliver targeted services
50:23
to farmers. So when we talk about data systems,
50:26
let's talk about the diversity they represent,
50:28
making sure that diversity captures the diversity
50:31
of farmers and production systems that we want
50:34
to deliver advisory services to, and to make
50:37
sure that we're getting the feedback so that
50:39
outputs from those advisory services are constantly
50:43
improved over time. And that in itself is a different
50:46
and diverse data set. So that's it for today.
50:50
We're going to dive into this in more detail.
50:53
See you on the next episode.