Latest / 6 Ps in AI Pods (AI6P) / 🗣️ AISW #124: Sumaiya Shrabony, USA-based Bangladeshi technical program manager

Transcript

🗣️ AISW #124: Sumaiya Shrabony, USA-based Bangladeshi technical program manager · 6 Ps in AI Pods (AI6P)

• 0:00
  SPEAKER_00: Hi, everyone.
• 0:00
  SPEAKER_00: This is Karen Smiley.
• 0:02
  SPEAKER_00: Welcome to 6P's in AI Pods.
• 0:07
  SPEAKER_00: This post is part of our AI 6P interview series on AI, software, and wetware.
• 0:13
  SPEAKER_00: Our guests share their experiences with using AI and how they feel
• 0:17
  SPEAKER_00: about AI using their data and content.
• 0:21
  SPEAKER_01: I'm delighted to welcome Sumayya Shagoni as my guest today on AI, software, and wetware.
• 0:27
  SPEAKER_01: Sumaya, thank you so much for joining me for this interview.
• 0:29
  SPEAKER_01: Please tell us about yourself, who you are, and what you do.
• 0:33
  SPEAKER_02: Hi Karen.
• 0:34
  SPEAKER_02: Thank you so much for inviting me on your podcast.
• 0:37
  SPEAKER_02: I'm Sumaya.
• 0:38
  SPEAKER_02: The short version of my background is that I run enterprise data and AI adoption at
• 0:43
  SPEAKER_02: a public research university in Colorado, United States.
• 0:47
  SPEAKER_02: But honestly, this short version does not really get at it.
• 0:51
  SPEAKER_02: Day-to-day,
• 0:51
  SPEAKER_02: I manage about 200 Azure data factory pipelines,
• 0:55
  SPEAKER_02: semantic models in SAS,
• 0:57
  SPEAKER_02: tabular Power BI dashboards across 10 plus departments,
• 1:01
  SPEAKER_02: and reliable security for over 530 plus users in a FERPA-governed environment.
• 1:07
  SPEAKER_02: So it means who can see what student data,
• 1:10
  SPEAKER_02: how that access is audited,
• 1:12
  SPEAKER_02: how we prove it when the compliance team asks, that is the work.
• 1:17
  SPEAKER_02: But what has happened since, I believe, January is a different story.
• 1:22
  SPEAKER_02: I started building with AI, not testing it, not piloting it, building with it.
• 1:27
  SPEAKER_02: I now run a 19-scale agentic system that researches and drafts my weekly content
• 1:34
  SPEAKER_02: across three platforms.
• 1:35
  SPEAKER_02: I built a five-agent job search pipeline that scans listings,
• 1:40
  SPEAKER_02: scores them, researches companies who sponsors H1B,
• 1:46
  SPEAKER_02: I have
• 1:51
  SPEAKER_02: a conference agent that writes speaker pitches.
• 1:55
  SPEAKER_02: I shipped two hackathon projects,
• 1:58
  SPEAKER_02: one on data readiness for ML, one on multimodal AI for humanitarian work.
• 2:05
  SPEAKER_02: Here's the thing that your listeners need to know about me is that I'm not someone
• 2:10
  SPEAKER_02: who just tried ChatGPT a few times and has opinions.
• 2:13
  SPEAKER_02: I'm someone who built her interprofessional infrastructure on top of AI
• 2:17
  SPEAKER_02: over roughly one and a half year.
• 2:20
  SPEAKER_02: And the more I built, the more complicated my feelings got about it.
• 2:25
  SPEAKER_02: I am an immigrant in the United States.
• 2:27
  SPEAKER_02: I came from Bangladesh at 19.
• 2:30
  SPEAKER_02: alone for a computer science degree.
• 2:32
  SPEAKER_02: And I'm also the eldest daughter of my family.
• 2:35
  SPEAKER_02: Both of those things turn out to matter a lot more for how I relate to AI
• 2:41
  SPEAKER_02: than my technical skills.
• 2:43
  SPEAKER_02: But I think we can get into that in the later questions.
• 2:47
  SPEAKER_01: Yes, thank you for sharing that background with us.
• 2:49
  SPEAKER_01: That's really interesting that you done all this with building these different skills.
• 2:53
  SPEAKER_01: And so, yeah, let's talk more about that.
• 2:55
  SPEAKER_01: Tell our audience a little bit about your level of experience with AI
• 2:59
  SPEAKER_01: and machine learning and analytics.
• 3:00
  SPEAKER_01: You've described some of the work that you've done with different AI tools,
• 3:04
  SPEAKER_01: but do you have formal education in it?
• 3:07
  SPEAKER_01: Do you use it professionally and personally?
• 3:10
  SPEAKER_01: It sounds like you do some of each.
• 3:12
  SPEAKER_02: Yes.
• 3:13
  SPEAKER_02: I do it both professionally and personally.
• 3:16
  SPEAKER_02: And for me, it sometimes feels like I'm more inside than is healthy.
• 3:22
  SPEAKER_02: Professionally,
• 3:23
  SPEAKER_02: I have spent roughly seven years in enterprise data analytics,
• 3:27
  SPEAKER_02: the architecture, data engineering, dimensional modeling, governance.
• 3:32
  SPEAKER_02: The AI part came in layers.
• 3:34
  SPEAKER_02: First, I joined our institutional AI working group at the university,
• 3:39
  SPEAKER_02: where we defined the guiding principles for how we are going
• 3:42
  SPEAKER_02: to roll out chat GPT-ADU across campus.
• 3:46
  SPEAKER_02: Privacy, transparency, human first explainability.
• 3:50
  SPEAKER_02: We wrote their principles,
• 3:51
  SPEAKER_02: then I had to actually implement that,
• 3:54
  SPEAKER_02: which is where the gap between policy and practice gets painfully real.
• 3:58
  SPEAKER_02: Personally is where it gets interesting because when I was doing my undergrad
• 4:03
  SPEAKER_02: and grad in computer science and information systems,
• 4:08
  SPEAKER_02: I did pick up machine learning courses.
• 4:11
  SPEAKER_02: Back in the day, it wasn't that much of AI application, it was more theory algorithms and such.
• 4:18
  SPEAKER_02: But nowadays when AI is so popular,
• 4:21
  SPEAKER_02: there's multiple tools in practice,
• 4:24
  SPEAKER_02: I decided I would just pick up some for my day-to-day use and my personal life.
• 4:29
  SPEAKER_02: So since January, I have been building agency AI systems.
• 4:34
  SPEAKER_02: I decided I will not just use AI as a tool,
• 4:37
  SPEAKER_02: but as a system that can be reused and be scheduled, even though I'm in the loop.
• 4:44
  SPEAKER_02: So I have that content system.
• 4:47
  SPEAKER_02: It has like 19 skills.
• 4:49
  SPEAKER_02: It researches the topic.
• 4:51
  SPEAKER_02: Drafts contents,
• 4:52
  SPEAKER_02: run voice complaint checks,
• 4:54
  SPEAKER_02: produce visual briefs,
• 4:55
  SPEAKER_02: cross-reference,
• 4:57
  SPEAKER_02: three different content calendars,
• 4:59
  SPEAKER_02: and handle platform-specific formatting for LinkedIn, Instagram, Substack.
• 5:04
  SPEAKER_02: My job search system is now most crucial for me since I'm in the job search phase.
• 5:09
  SPEAKER_02: So it has five agents,
• 5:11
  SPEAKER_02: a scanner, a scorer, a company researcher, a resume tailor, and an outreach writer.
• 5:17
  SPEAKER_02: They coordinate.
• 5:18
  SPEAKER_02: They also deduplicate.
• 5:19
  SPEAKER_02: They have their own daily schedule.
• 5:22
  SPEAKER_02: So I just want your listeners to sit with it for a second that I automated my own job search.
• 5:28
  SPEAKER_02: I automated my writing.
• 5:30
  SPEAKER_02: So I automated the thing that is supposed to represent my voice and the thing
• 5:34
  SPEAKER_02: that is supposed to represent my career.
• 5:37
  SPEAKER_02: And I did that because it works.
• 5:39
  SPEAKER_02: The system genuinely helps guide my chaotic thinking,
• 5:42
  SPEAKER_02: overwhelming workload and produces better output than I could do manually at
• 5:46
  SPEAKER_02: the same volume.
• 5:48
  SPEAKER_02: But this is the part I did not expect.
• 5:51
  SPEAKER_02: The more I automate, the less I recognize myself in the output.
• 5:56
  SPEAKER_02: And that's the tension.
• 5:57
  SPEAKER_02: I think I'm going to talk about a bit more in the later questions.
• 6:01
  SPEAKER_01: Yeah, that's a great observation that it is more productive,
• 6:04
  SPEAKER_01: but then it maybe is less you than what you would have written originally
• 6:09
  SPEAKER_01: for yourselves.
• 6:09
  SPEAKER_01: Can you tell us a specific story on how you are using a tool that has
• 6:14
  SPEAKER_01: or machine learning features?
• 6:15
  SPEAKER_01: I'd like to hear your thoughts about how well those features work for you,
• 6:19
  SPEAKER_01: basically what went well and maybe what didn't go so well.
• 6:22
  SPEAKER_02: I think I can give you a few.
• 6:26
  SPEAKER_02: As I mentioned,
• 6:27
  SPEAKER_02: AI shows up in my life at every level now, from the mundane to the deeply personal.
• 6:32
  SPEAKER_02: And the experience is different at each one.
• 6:36
  SPEAKER_02: So the everyday stuff first.
• 6:38
  SPEAKER_02: I use Grammarly with AI features at almost everything I write professionally
• 6:43
  SPEAKER_02: because English is not my first language.
• 6:46
  SPEAKER_02: I tried to make sure my grammar,
• 6:49
  SPEAKER_02: the style checking, everything is genuinely useful with Grammarly's AI feature.
• 6:54
  SPEAKER_02: It catches things I miss, especially when I'm writing fast.
• 6:57
  SPEAKER_02: Then Grammarly added this generative AI suggestions, and that is where it got very weird for me.
• 7:04
  SPEAKER_02: It started suggesting readouts that were technically smooth, but sounded nothing like me.
• 7:09
  SPEAKER_02: I would write a dark,
• 7:10
  SPEAKER_02: slightly rough sentence and Grammarly would offer a polished version
• 7:14
  SPEAKER_02: that read like consulting memo.
• 7:17
  SPEAKER_02: I started declining more and more those AI rewrites.
• 7:20
  SPEAKER_02: I don't think I have even accepted them anymore in recent months.
• 7:26
  SPEAKER_02: The tool was optimizing for a voice.
• 7:29
  SPEAKER_02: It's really not mine.
• 7:31
  SPEAKER_02: And I'm actively trying to avoid that, even though it feels so easy and fast to accept it.
• 7:39
  SPEAKER_02: The Zoom AI summaries, I see it in so many cross departmental meetings at the university.
• 7:45
  SPEAKER_02: So we have our Zoom enterprise plan and it started generating meeting summaries automatically.
• 7:52
  SPEAKER_02: And they're honestly about, I think 70% useful.
• 7:57
  SPEAKER_02: They captured the topics discussed,
• 7:59
  SPEAKER_02: the action items, the decisions, what they miss is the subtext.
• 8:03
  SPEAKER_02: They cannot capture that the provost's tone kind of shifted when we brought up
• 8:08
  SPEAKER_02: the chat GPT-EDU budget.
• 8:10
  SPEAKER_02: They cannot capture that three people went quiet when the data governance question came up.
• 8:16
  SPEAKER_02: The summary says discussed data governance approach.
• 8:20
  SPEAKER_02: And what actually happened was a 15-minute tension about who owns the AI policy.
• 8:25
  SPEAKER_02: So the summary is correct, but it's missing everything that matters.
• 8:29
  SPEAKER_02: So if we go by these summaries,
• 8:32
  SPEAKER_02: our decisions in the future will just keep going in the wrong direction.
• 8:38
  SPEAKER_02: And then the big one is my content system that I built for my content pipeline.
• 8:45
  SPEAKER_02: I have six refinement passes in the system.
• 8:48
  SPEAKER_02: One of them is called the skeptic, where it checks, could anyone have written this?
• 8:53
  SPEAKER_02: And if yes, then rewrite.
• 8:55
  SPEAKER_02: Another is called the competitor,
• 8:57
  SPEAKER_02: where it checks what angle would someone with 10x my followers already be posting?
• 9:03
  SPEAKER_02: And it works.
• 9:04
  SPEAKER_02: The system produces drafts that are structured on brand,
• 9:08
  SPEAKER_02: site, real sources, references, my actual technical stack.
• 9:13
  SPEAKER_02: But here is what really feels wrong.
• 9:16
  SPEAKER_02: And even though I try to optimize the prompts,
• 9:20
  SPEAKER_02: the reference files,
• 9:21
  SPEAKER_02: the content pillars,
• 9:23
  SPEAKER_02: it just I think your listeners will recognize it even in a bigger or smaller scale.
• 9:29
  SPEAKER_02: On February, I kind of got fed up with my outputs.
• 9:32
  SPEAKER_02: It was incredible, the essays.
• 9:35
  SPEAKER_02: I will not say that it was bad.
• 9:37
  SPEAKER_02: It just felt too incredible.
• 9:40
  SPEAKER_02: It wasn't me at all in many cases because the structure was right,
• 9:44
  SPEAKER_02: but I was just feeling like I'm missing my soul because It has no hesitation.
• 9:50
  SPEAKER_02: It's way too confident in its writing.
• 9:53
  SPEAKER_02: It just doesn't say to my readers what can go wrong, where I lack confidence.
• 10:00
  SPEAKER_02: So now when my agent runs on a Sunday, I have this ritual.
• 10:06
  SPEAKER_02: I will sit with my draft and put the rough edges back in.
• 10:10
  SPEAKER_02: I will add my hesitations.
• 10:12
  SPEAKER_02: I will break the clean sentences.
• 10:15
  SPEAKER_02: Isn't it weird to do this with your Sunday?
• 10:17
  SPEAKER_02: I would never have done it if I didn't have an agent system, you know?
• 10:22
  SPEAKER_02: So yeah, the tools are powerful enough to produce output that passes.
• 10:27
  SPEAKER_02: Grammar check pass,
• 10:28
  SPEAKER_02: meeting summaries pass,
• 10:30
  SPEAKER_02: first drafts pass, but passes is actually not the same as is right.
• 10:35
  SPEAKER_02: And the gap between passing and being right is where the human work actually lives now.
• 10:41
  SPEAKER_02: So your listeners who use any AI feature tool,
• 10:44
  SPEAKER_02: even just Grammarly or Zoom are navigating that gap every day,
• 10:48
  SPEAKER_02: whether they have named it or not.
• 10:51
  SPEAKER_01: Yeah.
• 10:51
  SPEAKER_01: And I think you're echoing things I've seen written on subset where we both hang out
• 10:55
  SPEAKER_01: and where we met about concerns that the systems go to the average
• 11:02
  SPEAKER_01: and they are averaging out your voice and yet voice is and should be distinctive.
• 11:07
  SPEAKER_01: That distinctive voice is what your subscribers want.
• 11:12
  SPEAKER_01: There's some risk of losing that if you let it sand away all the edges
• 11:16
  SPEAKER_01: and smooth everything out and make it average.
• 11:19
  SPEAKER_02: Yeah.
• 11:20
  SPEAKER_02: And sometimes if you keep seeing more outputs like this,
• 11:24
  SPEAKER_02: you kind of even forget whether it is your voice or whether this is what it used
• 11:30
  SPEAKER_02: to be.
• 11:31
  SPEAKER_01: It sounds like you are making good use of AI tools for a lot of different purposes.
• 11:35
  SPEAKER_01: Are there any things that you avoid using AI tools for something
• 11:40
  SPEAKER_01: that you would never use AI for?
• 11:42
  SPEAKER_01: And so can you share an example of when and why you would choose not to use it?
• 11:47
  SPEAKER_02: Yes.
• 11:47
  SPEAKER_02: So there are some ongoing roles that I set for myself and
• 11:51
  SPEAKER_02: a few more specific moments where I actively walked away from a tool
• 11:55
  SPEAKER_02: that was right there, ready to go.
• 11:58
  SPEAKER_02: And so I tried using Generative AI for technical architecture diagrams early on.
• 12:04
  SPEAKER_02: I needed visual showing of how our models connect to Power BI through
• 12:10
  SPEAKER_02: the XMLA endpoint with row-level security filtering at the semantic layer.
• 12:15
  SPEAKER_02: The AI-generated diagram looked professional.
• 12:18
  SPEAKER_02: Clean boxes, clean arrows, and the data flow was wrong.
• 12:22
  SPEAKER_02: It had Power BI querying the SQL Server directly instead of going through the semantic layer.
• 12:28
  SPEAKER_02: And the more I tried to correct that visual, the more it started messing up.
• 12:36
  SPEAKER_02: And I realized that if I had published that,
• 12:39
  SPEAKER_02: someone on my team or someone reading my newsletter could have built something based
• 12:44
  SPEAKER_02: on a structural lie.
• 12:46
  SPEAKER_02: And a wrong architecture diagram is not a minor mistake.
• 12:50
  SPEAKER_02: It is a confident integration of how a system actually works.
• 12:55
  SPEAKER_02: After that,
• 12:55
  SPEAKER_02: I decided I'm never going to spend or waste that much of my time in generative AI,
• 13:01
  SPEAKER_02: at least for technical architecture diagrams.
• 13:04
  SPEAKER_02: I now specifically use Excalibur Draw and draw the diagrams with my hand.
• 13:10
  SPEAKER_02: It's not handled like the Excalibur app, but it's just the way it is.
• 13:15
  SPEAKER_02: I cannot go back.
• 13:16
  SPEAKER_02: Even though I know now ChatGPT is way better, but I just cannot.
• 13:21
  SPEAKER_02: I cannot trust it to give me the better diagram that I can do manually.
• 13:27
  SPEAKER_02: And the ongoing rule for me is governance, especially in my professional work.
• 13:34
  SPEAKER_02: At CU Denver, where I work, and I manage role level security for the departments.
• 13:40
  SPEAKER_02: It means who can see which student data when and under what conditions.
• 13:44
  SPEAKER_02: I have been asked more than once whether we could use AI to help with access decisions.
• 13:51
  SPEAKER_02: Automate the review process, flag anomalies, and I use AI for the auditing side.
• 13:56
  SPEAKER_02: Absolutely.
• 13:57
  SPEAKER_02: Pattern detection, compliance checks, and this is what machines are good at.
• 14:02
  SPEAKER_02: But the actual decision, should this person see the data?
• 14:06
  SPEAKER_02: I will not automate that.
• 14:07
  SPEAKER_02: Because it is not a classification problem.
• 14:09
  SPEAKER_02: It is a values problem.
• 14:11
  SPEAKER_02: It involves institutional trust.
• 14:13
  SPEAKER_02: Why it is?
• 14:14
  SPEAKER_02: Because historical relationships between departments,
• 14:17
  SPEAKER_02: FERPA compliance,
• 14:18
  SPEAKER_02: where the stakes are real,
• 14:20
  SPEAKER_02: the kind of conceptual judgment that a model trained on,
• 14:24
  SPEAKER_02: general patterns cannot make.
• 14:26
  SPEAKER_02: The model does not know that the reason I restricted a specific report last year was
• 14:31
  SPEAKER_02: a conversation between two directors.
• 14:34
  SPEAKER_02: That never made it into any system.
• 14:37
  SPEAKER_02: That context leaves on people.
• 14:40
  SPEAKER_02: And I cannot expect my system to really know this answer because it was never in any
• 14:47
  SPEAKER_02: of the system, right?
• 14:48
  SPEAKER_02: It wasn't written down.
• 14:49
  SPEAKER_02: So it's not the system's fault.
• 14:51
  SPEAKER_02: It's just I'm never gonna trust it.
• 14:53
  SPEAKER_02: With the small nuances or the context that just leaves the people.
• 14:59
  SPEAKER_02: There was one more that I started doing that will be resonated by many people.
• 15:04
  SPEAKER_02: I have been reading about it in a couple of my friends' posts too,
• 15:08
  SPEAKER_02: that people have stopped using AI-generated replies in emails.
• 15:14
  SPEAKER_02: I have stopped it too,
• 15:15
  SPEAKER_02: like in Gmail,
• 15:16
  SPEAKER_02: Outlook or whatever platforms,
• 15:18
  SPEAKER_02: I will just turn off the suggested replies because I noticed
• 15:22
  SPEAKER_02: that I was just clicking sounds good to be done with one of the replies
• 15:27
  SPEAKER_02: since it's fast,
• 15:29
  SPEAKER_02: but then I was putting replies that sounds good to the emails
• 15:35
  SPEAKER_02: that just doesn't sound good.
• 15:37
  SPEAKER_02: I was trying to avoid friction, but it's just making my tone go off.
• 15:43
  SPEAKER_02: Or since I'm just outsourcing this,
• 15:46
  SPEAKER_02: it's just making me kind of inactive
• 15:49
  SPEAKER_03: from all the bigger decision makings.
• 15:52
  SPEAKER_03: And that might really affect my career trajectories in the long run.
• 15:58
  SPEAKER_03: I think for your listeners, the question is not, should I avoid AI?
• 16:02
  SPEAKER_03: It's more like, do I notice when AI is making decisions I did not realize I was delegating?
• 16:09
  SPEAKER_03: The suggested reply is a decision about your tone.
• 16:12
  SPEAKER_03: The AI-generated summary is a decision about what mattered in a meeting.
• 16:17
  SPEAKER_02: The auto-generated architecture diagram is a decision about how your system works.
• 16:22
  SPEAKER_02: If you're not checking those decisions, you're not using a tool.
• 16:25
  SPEAKER_02: The tool is using your name.
• 16:28
  SPEAKER_01: Yeah, very important distinction.
• 16:30
  SPEAKER_01: I'm glad you called that out.
• 16:31
  SPEAKER_01: I'm going to switch gears a little bit and talk a bit about what's behind these systems.
• 16:36
  SPEAKER_01: One growing concern is where AI and ML systems get the data and the works that they train on.
• 16:43
  SPEAKER_00: A lot of times
• 16:44
  SPEAKER_01: they're using data that people put into online systems
• 16:47
  SPEAKER_01: or publish online,
• 16:49
  SPEAKER_01: and the companies are not always transparent about how they intend to use our data
• 16:53
  SPEAKER_01: when we sign up and agree to share it with them.
• 16:55
  SPEAKER_01: I'm wondering how you feel about companies using people's data,
• 17:00
  SPEAKER_01: people's works for training their tools.
• 17:03
  SPEAKER_01: One question that comes up is whether they should be required
• 17:07
  SPEAKER_01: to get people's consent to give them credit or to compensate them.
• 17:12
  SPEAKER_01: We're all three.
• 17:13
  SPEAKER_01: Some people have thoughts about that, about whether it's a good thing or a bad thing.
• 17:16
  SPEAKER_01: And I'd like to hear your thoughts.
• 17:18
  SPEAKER_02: Well, let's not talk about good thing or bad thing first.
• 17:22
  SPEAKER_02: Let me tell you my answer from two different positions because I think I hold both
• 17:27
  SPEAKER_02: at the same time and the tension between them is the honest answer.
• 17:32
  SPEAKER_02: My number one position is from my day job where I'm a data governance practitioner, right?
• 17:38
  SPEAKER_02: So at work,
• 17:39
  SPEAKER_02: when I'm managing access to student data under FERPA,
• 17:44
  SPEAKER_02: Department wants to see enrollment numbers.
• 17:47
  SPEAKER_02: There is a process.
• 17:48
  SPEAKER_02: There is documentation.
• 17:49
  SPEAKER_02: There is a review.
• 17:50
  SPEAKER_02: I can tell you who has access to what and why and when it was last audited.
• 17:55
  SPEAKER_02: We do this because the data belongs to students, not to us.
• 17:59
  SPEAKER_02: That principle is non-negotiable in my professional life.
• 18:05
  SPEAKER_02: In my second position, I'm a heavy AI user, right?
• 18:09
  SPEAKER_02: I'm benefiting every single day from models that were trained
• 18:13
  SPEAKER_02: on data their creators never consented to.
• 18:16
  SPEAKER_02: So even though my content system runs on cloud code,
• 18:20
  SPEAKER_02: my job search agent runs on codex,
• 18:23
  SPEAKER_02: the value I extract from those tools exist because someone's writing,
• 18:27
  SPEAKER_02: someone's code, someone's published work became training data without anyone asking.
• 18:33
  SPEAKER_02: I hold both of these positions simultaneously.
• 18:36
  SPEAKER_02: And I think anyone who is honest about their relationship with AI does too.
• 18:43
  SPEAKER_02: And I feel contradiction about both of my feelings.
• 18:47
  SPEAKER_02: In one case,
• 18:48
  SPEAKER_02: I'm holding governance in such a strict standard,
• 18:52
  SPEAKER_02: but in another way, I am actually benefiting from the abuse of that.
• 19:01
  SPEAKER_02: I want to be specific about why each one matters,
• 19:04
  SPEAKER_02: consent, credit,
• 19:04
  SPEAKER_02: and compensation,
• 19:06
  SPEAKER_02: because I think people collapse them into one thing,
• 19:09
  SPEAKER_02: and there are actually three different problems.
• 19:11
  SPEAKER_02: Consent is the foundation.
• 19:14
  SPEAKER_02: If you read Monika's 3C framework,
• 19:17
  SPEAKER_02: you know that this was originally for protecting indigenous culture property,
• 19:22
  SPEAKER_02: and the principle transfers perfectly.
• 19:25
  SPEAKER_02: Prior and informed consent,
• 19:27
  SPEAKER_02: not buried in paragraph 47 of the Terms of Service,
• 19:31
  SPEAKER_02: not opt-in by default with a toggle you did not know existed,
• 19:36
  SPEAKER_02: actual consent,
• 19:37
  SPEAKER_02: the kind word the person understands what they are agreeing to
• 19:40
  SPEAKER_02: before they even agree.
• 19:42
  SPEAKER_02: Then there's credit.
• 19:43
  SPEAKER_02: It matters because invisible labor is exploitable labor.
• 19:48
  SPEAKER_02: When a musician's work, trains Suno or Udiyo.
• 19:51
  SPEAKER_02: When a writer's essays train a language model.
• 19:54
  SPEAKER_02: When a software's code trains co-pilot,
• 19:57
  SPEAKER_02: the output comes with the person whose work made it possible.
• 20:01
  SPEAKER_02: But the person is invisible, no attribution, no acknowledgement, no record.
• 20:07
  SPEAKER_02: Credit does not solve the economic problem,
• 20:09
  SPEAKER_02: but it makes the relationship visible and visibility is where accountability starts.
• 20:15
  SPEAKER_02: And compensation is the hardest one because I think this is
• 20:18
  SPEAKER_02: the one companies will resist the longest.
• 20:21
  SPEAKER_02: Because compensation means the training data has a cost now.
• 20:25
  SPEAKER_02: And right now,
• 20:26
  SPEAKER_02: the entire economic model of foundational models depend on training data being free.
• 20:32
  SPEAKER_02: If you had to compensate every writer,
• 20:35
  SPEAKER_02: every artist,
• 20:36
  SPEAKER_02: every developer whose work trained GPT-4 or Cloud or Gemini,
• 20:40
  SPEAKER_02: the cost structure changes fundamentally.
• 20:43
  SPEAKER_02: I think it should change,
• 20:45
  SPEAKER_02: but I also think I need to be honest that I benefit from
• 20:48
  SPEAKER_02: the current arrangement every day.
• 20:51
  SPEAKER_02: So if you have ever written a blog post,
• 20:54
  SPEAKER_02: published a photo,
• 20:56
  SPEAKER_02: posted code on GitHub,
• 20:58
  SPEAKER_02: left a review,
• 20:59
  SPEAKER_02: you are a data contributor to those models, whether you know it or not.
• 21:03
  SPEAKER_02: Even if I Feel is not good.
• 21:07
  SPEAKER_02: Heart is bad.
• 21:08
  SPEAKER_02: The supply chain is there.
• 21:10
  SPEAKER_02: And I'm also the problem inside the supply chain.
• 21:14
  SPEAKER_01: Yeah.
• 21:14
  SPEAKER_01: I think a lot of people are wrestling with that duality and I appreciate you being
• 21:19
  SPEAKER_01: so candid about seeing it from both sides.
• 21:22
  SPEAKER_01: So thank you for sharing that.
• 21:24
  SPEAKER_01: As someone who uses tools,
• 21:25
  SPEAKER_01: you mentioned looking at terms and conditions and page 47 and not being aware of it.
• 21:31
  SPEAKER_01: Do you feel like any of the providers or the tools
• 21:33
  SPEAKER_01: that you use have been transparent with you about where they got the data
• 21:38
  SPEAKER_01: that they use for training their models and whether there was consent from
• 21:42
  SPEAKER_01: the people whose data it is?
• 21:45
  SPEAKER_02: No.
• 21:46
  SPEAKER_02: And I think it has been a escalating pattern since 2024 and 2026 kind of made it worse.
• 21:53
  SPEAKER_02: The most recent example that genuinely shocked me is GitHub because I use this extensively.
• 22:00
  SPEAKER_02: In March 2026,
• 22:01
  SPEAKER_02: they announced that starting April 24 interaction data from Copilot Free Pro
• 22:07
  SPEAKER_02: and ProPlus users,
• 22:09
  SPEAKER_02: your inputs,
• 22:10
  SPEAKER_02: your outputs,
• 22:11
  SPEAKER_02: your core snippets, all of it would be used to train AI models unless you opt out.
• 22:16
  SPEAKER_02: Opt out, not opt in.
• 22:18
  SPEAKER_02: Default is opt in.
• 22:21
  SPEAKER_02: Your code that trains their model is opt-in by default.
• 22:26
  SPEAKER_02: The community discussion on GitHub got so many downboards and responses,
• 22:32
  SPEAKER_02: but only one GitHub VP defended the plan.
• 22:35
  SPEAKER_02: Developers were furious and this is GitHub, a platform where people publish open source code.
• 22:41
  SPEAKER_02: Even that community which understands licensing better
• 22:44
  SPEAKER_02: than almost any user base felt blindsided by the default.
• 22:49
  SPEAKER_02: Then there's Atlassian,
• 22:50
  SPEAKER_02: which just announced that starting August,
• 22:52
  SPEAKER_02: they will use customer data from Jira,
• 22:55
  SPEAKER_02: Confluence, and other cloud products to train Robo and RoboDev.
• 23:00
  SPEAKER_02: 300,000 customers were affected.
• 23:02
  SPEAKER_02: And here is the part that should alarm anyone who uses enterprise SaaS.
• 23:06
  SPEAKER_02: If you're on free, standard, or premium tiers, metadata collection is mandatory.
• 23:11
  SPEAKER_02: You cannot opt out.
• 23:13
  SPEAKER_02: Only enterprise customers get opt-out controls.
• 23:16
  SPEAKER_02: So Atlassian created a system where opting out of AI training is
• 23:21
  SPEAKER_02: a premium feature you have to pay for.
• 23:24
  SPEAKER_02: And this reverses their prior stated position that customer data would not be used
• 23:29
  SPEAKER_02: for AI training.
• 23:30
  SPEAKER_02: They said one thing, they're doing the opposite.
• 23:34
  SPEAKER_02: That is not a transparency gap.
• 23:36
  SPEAKER_02: That is a broken promise with a price tag.
• 23:39
  SPEAKER_02: And I think X is the most extreme case right now.
• 23:43
  SPEAKER_02: Their January Terms Revision explicitly treats AI prompts,
• 23:47
  SPEAKER_02: outputs,
• 23:48
  SPEAKER_02: and interactions as content under a worldwide royalty-free sub-licensable license
• 23:54
  SPEAKER_02: for any purpose, including training.
• 23:56
  SPEAKER_02: And the opt-out tool provides incomplete protection
• 24:00
  SPEAKER_02: because your data remains available to X for other purposes under the 2026 terms.
• 24:06
  SPEAKER_02: That is not an opt-out.
• 24:07
  SPEAKER_02: That is complete theater.
• 24:11
  SPEAKER_01: So, Somalia,
• 24:12
  SPEAKER_01: when you've worked with building AI-based tools or systems,
• 24:15
  SPEAKER_01: can you share about where the data came from and how it was obtained?
• 24:19
  SPEAKER_02: Yeah, sure.
• 24:20
  SPEAKER_02: So, yeah,
• 24:21
  SPEAKER_02: I have built AI systems,
• 24:23
  SPEAKER_02: my content system, the job search pipeline and hackathon projects.
• 24:26
  SPEAKER_02: And here I can tell you about where the data came from because I Put it there
• 24:32
  SPEAKER_02: and I know where it came from, right?
• 24:33
  SPEAKER_02: My content system does not fine-tune a model.
• 24:36
  SPEAKER_02: It uses Cloud as a runtime and fears it in my context at interference time.
• 24:42
  SPEAKER_02: The training data for my system is my own voice rules,
• 24:45
  SPEAKER_02: my own content pillars,
• 24:47
  SPEAKER_02: my own brand guidelines, my own editorial standards, files I wrote.
• 24:52
  SPEAKER_02: My hackathon project, Data Ready, works with email data sets to assess data readiness.
• 24:57
  SPEAKER_02: The data sets it evaluates are user provided and is never stored in any cloud platforms.
• 25:02
  SPEAKER_02: It also uses public benchmark sets to give you the demo scores.
• 25:08
  SPEAKER_02: I think this distinction matters because there is a difference between building
• 25:12
  SPEAKER_02: a system that uses an existing model's capabilities with your own data
• 25:18
  SPEAKER_02: at interference time and building a system that scrapes other people's work to train
• 25:23
  SPEAKER_02: a new model.
• 25:24
  SPEAKER_02: I do the first, but the companies whose models I build on did the second.
• 25:28
  SPEAKER_02: So I'm benefiting from both,
• 25:30
  SPEAKER_02: which is why I do not think anyone gets to be purely self-righteous about this.
• 25:36
  SPEAKER_02: As I mentioned, the supply chain is compromised and I'm actively in the supply chain.
• 25:41
  SPEAKER_01: Yeah, that's a very fair point.
• 25:43
  SPEAKER_01: And you've probably seen these articles as well.
• 25:45
  SPEAKER_01: If you look farther back in the supply chain too,
• 25:48
  SPEAKER_01: how the data has been processed and where we get the devices that we use and where
• 25:53
  SPEAKER_01: the minerals come from to build them.
• 25:54
  SPEAKER_01: That entire supply chain going far back beyond what most people even see or think
• 25:59
  SPEAKER_01: about,
• 26:00
  SPEAKER_01: it's all compromised too.
• 26:01
  SPEAKER_01: There's a great article by Rebecca that I just read talking
• 26:05
  SPEAKER_01: about how being ethical may mean accepting that we have a responsibility for this,
• 26:12
  SPEAKER_01: that there's no way to make a perfectly ethical, innocent choice.
• 26:17
  SPEAKER_01: In some cases, there's just no good options.
• 26:20
  SPEAKER_01: We don't have better options.
• 26:21
  SPEAKER_01: And I think a lot of us are trying to fight so that we have better options,
• 26:25
  SPEAKER_01: so we have more ethical tools, but that's not where we are right now.
• 26:30
  SPEAKER_02: Yeah, exactly.
• 26:31
  SPEAKER_02: Even if I use local AI models, there's still no 100% better way to make it fully ethical.
• 26:42
  SPEAKER_02: That's what I found out.
• 26:44
  SPEAKER_01: Yeah.
• 26:44
  SPEAKER_01: But we can still try.
• 26:45
  SPEAKER_00: Yeah, obviously we have
• 26:47
  SPEAKER_01: to try.
• 26:48
  SPEAKER_01: Yes.
• 26:49
  SPEAKER_01: There are things that we can do and we can take steps.
• 26:51
  SPEAKER_01: I'm always glad to hear when people are making efforts.
• 26:53
  SPEAKER_01: Let's talk about your data for a minute.
• 26:55
  SPEAKER_01: Just being consumers,
• 26:57
  SPEAKER_01: being members of the public,
• 26:59
  SPEAKER_01: our personal data and our content has probably been used by some AI-based tools
• 27:04
  SPEAKER_01: or systems.
• 27:05
  SPEAKER_01: Do you know of any cases that you could share when your data has been used?
• 27:10
  SPEAKER_02: Yes, I can probably share three at different scales because I think
• 27:15
  SPEAKER_02: the range is what makes this real for the listeners.
• 27:18
  SPEAKER_02: The personal one is I have been active on LinkedIn for I think almost 10 years now.
• 27:24
  SPEAKER_02: I post three to four times a week, technical analysis, career reflection, enterprise frameworks.
• 27:31
  SPEAKER_02: LinkedIn has been using member content to train their AI models
• 27:35
  SPEAKER_02: since I believe 2020 opted in by default, opt out only by prospective.
• 27:42
  SPEAKER_02: But here's what happened more recently.
• 27:44
  SPEAKER_02: In March, 2026, I mentioned it before too.
• 27:47
  SPEAKER_02: GitHub announced the same thing for Copilot users.
• 27:50
  SPEAKER_02: Your code snippets,
• 27:51
  SPEAKER_02: your inputs, your outputs, training their models starting April, unless you opt out.
• 27:56
  SPEAKER_02: I use Copilot.
• 27:57
  SPEAKER_02: I write code.
• 27:58
  SPEAKER_02: I did not consent.
• 28:00
  SPEAKER_02: I found out from a news article, went into settings and toggled the opt out.
• 28:04
  SPEAKER_03: But here is the question I cannot answer.
• 28:06
  SPEAKER_03: Was my code already used before I toggled?
• 28:10
  SPEAKER_03: GitHub says previously opted out, preferences are retained.
• 28:14
  SPEAKER_03: But what are the period between this started collecting and when I found out?
• 28:19
  SPEAKER_03: That gap is where trust completely dies.
• 28:22
  SPEAKER_03: And then Atlassian, right?
• 28:24
  SPEAKER_03: It's heavily used in my work.
• 28:27
  SPEAKER_03: We use Jira and Confluence at work.
• 28:29
  SPEAKER_03: We manage sprints, document architecture decisions.
• 28:32
  SPEAKER_03: We track incidents.
• 28:33
  SPEAKER_03: They announced that starting August, they will use the data to train Rovo.
• 28:38
  SPEAKER_03: And if we are not on enterprise tier, metadata collection is mandatory.
• 28:42
  SPEAKER_03: We cannot opt out even if we want to without paying extra.
• 28:45
  SPEAKER_03: And sprint velocity data.
• 28:47
  SPEAKER_02: Our issue descriptions,
• 28:49
  SPEAKER_02: our workflow names, all of it fits their AI whether we agree to it or not.
• 28:53
  SPEAKER_02: This is enterprise data.
• 28:55
  SPEAKER_02: Some of it is FERPA-edjacent in my environment.
• 28:59
  SPEAKER_02: The idea that a SaaS vendor can unilaterally decide to train AI models
• 29:04
  SPEAKER_02: on my institutional workflow data and then the opt-out is
• 29:09
  SPEAKER_02: a paid premium feature should concern anyone who manages data in
• 29:13
  SPEAKER_02: a regulated environment.
• 29:15
  SPEAKER_02: The one that sits with me most,
• 29:17
  SPEAKER_02: honestly,
• 29:17
  SPEAKER_02: is watching my own content come back to me in someone else's mouth, you know.
• 29:23
  SPEAKER_02: As I write weekly newsletters,
• 29:25
  SPEAKER_02: Enterprise AI Adoption from practitioners said, I put hours into each one.
• 29:30
  SPEAKER_02: Even though I'm using a content system, all of my contents are my own experiences.
• 29:36
  SPEAKER_02: Whenever I'm finding my technical references, I sit with the drafts for hours to make it.
• 29:44
  SPEAKER_02: Then I will see a generated LinkedIn post on the same topics,
• 29:48
  SPEAKER_02: sometimes within days or hours using the same framing.
• 29:52
  SPEAKER_02: Same structural argument, sometimes even the same example sequence.
• 29:56
  SPEAKER_02: I cannot prove those posts were trained on my writing specifically, right?
• 30:01
  SPEAKER_02: But I publish publicly,
• 30:02
  SPEAKER_02: my work is indiced, and language models do not spontaneously arrive at the same.
• 30:07
  SPEAKER_02: Here is how role level security governance fills in multi-department power
• 30:11
  SPEAKER_02: by deployment framing that I wrote last month, right?
• 30:15
  SPEAKER_02: So the economics of it are what bothers me.
• 30:18
  SPEAKER_02: I am not a full-time paid writer.
• 30:20
  SPEAKER_02: So this is not a direct income loss for me,
• 30:23
  SPEAKER_02: but I watched the effort to reach collapse in real time.
• 30:26
  SPEAKER_02: Someone can prompt a model with write a LinkedIn post
• 30:29
  SPEAKER_02: about enterprise BI governance challenges,
• 30:32
  SPEAKER_02: get a version that is structurally similar to what I spent four hours writing posted
• 30:38
  SPEAKER_02: in 90 seconds.
• 30:39
  SPEAKER_02: Get comparable engagement because the algorithm does not distinguish between
• 30:43
  SPEAKER_02: a four hour effort and a four minute effort.
• 30:47
  SPEAKER_02: The content I make is being used at some level to train the models that produce
• 30:52
  SPEAKER_02: the content that compares with mine.
• 30:55
  SPEAKER_02: And there is no mechanism for me to know to consent or to be compensated.
• 31:01
  SPEAKER_02: It's not hypothetical.
• 31:02
  SPEAKER_02: It's just a normal Tuesday note.
• 31:04
  SPEAKER_02: So I want to name something about the opt out conversation that I think gets missed.
• 31:10
  SPEAKER_02: When people say just opt out, they assume that opting out is free.
• 31:14
  SPEAKER_02: It's not.
• 31:15
  SPEAKER_02: In my scenario,
• 31:17
  SPEAKER_02: as I'm on an immigrant visa,
• 31:19
  SPEAKER_02: every time I push back on a system,
• 31:21
  SPEAKER_02: I'm aware that I'm making a choice that carries a different weight than it does for
• 31:25
  SPEAKER_02: a citizen.
• 31:26
  SPEAKER_02: Opting out of LinkedIn AI training means accepting that my content gets less distribution.
• 31:33
  SPEAKER_02: Which means fewer recruiter views,
• 31:35
  SPEAKER_02: which matters when my visa status completely depends on employment.
• 31:39
  SPEAKER_03: Challenging a vendor's data policy at
• 31:41
  SPEAKER_02: work means being the person
• 31:43
  SPEAKER_02: who slows down adoption,
• 31:45
  SPEAKER_02: which is a different risk when you are not a permanent employee category because
• 31:49
  SPEAKER_02: of your visa status.
• 31:51
  SPEAKER_02: So the costing of opting out is not evenly distributed,
• 31:55
  SPEAKER_02: and the people with the least institutional power pay the highest price
• 31:59
  SPEAKER_02: of saying no.
• 32:02
  SPEAKER_01: Having a higher risk as a non-citizen.
• 32:04
  SPEAKER_01: One example that a lot of people bring up is going through TSA when they travel.
• 32:09
  SPEAKER_01: And there's a note on the TSA signs at the bottom in small printing
• 32:13
  SPEAKER_01: that you are entitled to opt out of allowing them to take a picture of your face
• 32:18
  SPEAKER_01: and use facial recognition on it.
• 32:20
  SPEAKER_01: But even for citizens, that can be an intimidating thing to end up and request that.
• 32:26
  SPEAKER_01: I understand it would be even more so for anyone who is not in the same legal status
• 32:31
  SPEAKER_01: and might feel less safe to do so.
• 32:33
  SPEAKER_01: As you said, it's not always free.
• 32:36
  SPEAKER_01: It's not always even any choice.
• 32:39
  SPEAKER_00: Yeah.
• 32:40
  SPEAKER_01: So that's an important consideration.
• 32:42
  SPEAKER_01: I think you've mentioned Atlassian and LinkedIn and it sounds like Atlassian
• 32:46
  SPEAKER_01: at least was proactive telling people about it before they did it.
• 32:50
  SPEAKER_01: But if you're on an enterprise system,
• 32:53
  SPEAKER_01: or if you're not on an enterprise system,
• 32:54
  SPEAKER_01: but using one of their pro licenses,
• 32:56
  SPEAKER_01: it's not like you can just change over your whole company's operation to
• 32:59
  SPEAKER_01: an enterprise plan within a short period of time.
• 33:02
  SPEAKER_01: Then I remember that in 2024, that they were not proactive about telling us.
• 33:07
  SPEAKER_01: They basically opted us in for everything up to that point.
• 33:11
  SPEAKER_01: And only the people in Europe got a choice about whether or not their content was used.
• 33:16
  SPEAKER_01: So they definitely didn't make us aware proactively.
• 33:20
  SPEAKER_01: Do you know of any other companies that have made you aware
• 33:22
  SPEAKER_01: that they would use your information for training an AI system?
• 33:28
  SPEAKER_02: That GitHub posts that one viral dimension upfront that their policy is changing.
• 33:34
  SPEAKER_02: It also discloses that individual users writing is used for product environment
• 33:39
  SPEAKER_02: that toggle exists.
• 33:41
  SPEAKER_02: But it's enabled by default.
• 33:43
  SPEAKER_02: So the majority of users are contributing to their writing model improvement
• 33:48
  SPEAKER_02: without ever knowing the setting exists.
• 33:50
  SPEAKER_02: Even for me, I actually didn't know that Grammarly's toggle is opt-in by default.
• 33:56
  SPEAKER_02: I had to go into their settings to check out,
• 33:59
  SPEAKER_02: I think after months of usage,
• 34:01
  SPEAKER_02: when I remembered, oh, let me check whether it's actually opt-in by default or not.
• 34:05
  SPEAKER_02: So this is what is surprising for me.
• 34:08
  SPEAKER_02: Why?
• 34:10
  SPEAKER_02: It has to be by default opt-in.
• 34:12
  SPEAKER_02: Why cannot it have opt-out by default?
• 34:14
  SPEAKER_02: Then give us an option.
• 34:15
  SPEAKER_02: Okay, please click to opt-in.
• 34:17
  SPEAKER_01: Yeah.
• 34:18
  SPEAKER_01: I think the short answer comes down to in some ways it's their business model.
• 34:23
  SPEAKER_01: They're counting on the fact that most people won't think to go in and look for things.
• 34:29
  SPEAKER_01: As soon as I create a new account somewhere,
• 34:31
  SPEAKER_01: I immediately go find the privacy settings,
• 34:34
  SPEAKER_01: but we shouldn't have to do that.
• 34:36
  SPEAKER_02: Exactly.
• 34:37
  SPEAKER_02: Like why should it be so hidden, especially for users who are not that technical, right?
• 34:43
  SPEAKER_02: People who doesn't have that awareness that these kinds of things can be completely plugged in.
• 34:49
  SPEAKER_02: They might not even realize that there are this kind of options or
• 34:53
  SPEAKER_02: the data is being used in this way.
• 34:55
  SPEAKER_02: For example,
• 34:56
  SPEAKER_02: my family back home,
• 34:58
  SPEAKER_02: they don't realize that if you use social media or LinkedIn,
• 35:02
  SPEAKER_02: the data is completely by default training their AI model unless they go inside
• 35:08
  SPEAKER_02: and opt out,
• 35:11
  SPEAKER_02: especially if you're not a primary English user,
• 35:14
  SPEAKER_02: but you're heavily using apps that are English based.
• 35:19
  SPEAKER_01: It's one thing to read the terms and conditions when we sign up,
• 35:22
  SPEAKER_01: but they also have been known to change them after the fact.
• 35:25
  SPEAKER_01: Sometimes there's a surprise, as you've mentioned, that sometimes they tell us in advance.
• 35:30
  SPEAKER_02: I want to be really specific because I think there is
• 35:34
  SPEAKER_02: a legal shift happening right now that the listeners should know about.
• 35:38
  SPEAKER_02: California's AB 2013 took effect January 2026.
• 35:42
  SPEAKER_02: It is the first US law that requires AI developers
• 35:46
  SPEAKER_02: to publicly disclose what data they use to train their models,
• 35:50
  SPEAKER_02: whether copyrighted materials were included,
• 35:53
  SPEAKER_02: and whether personal information was part of the training set.
• 35:56
  SPEAKER_02: That is new,
• 35:57
  SPEAKER_02: and on August 2026,
• 35:59
  SPEAKER_02: the EU AI Act's enforcement provisions for high-risk systems
• 36:03
  SPEAKER_02: and general-purpose AI models become enforceable,
• 36:06
  SPEAKER_02: with fines up to €15 million or 3% of global revenue.
• 36:11
  SPEAKER_02: That includes mandatory training data disclosures tied to the EU Copyright Directive opt-out.
• 36:17
  SPEAKER_02: So the regulatory landscape is shifting,
• 36:19
  SPEAKER_02: but right now for a US-based user like me or EU,
• 36:23
  SPEAKER_02: the answer to do I have a real choice is it depends on my geography,
• 36:28
  SPEAKER_02: my subscription tier, and how carefully I read the settings page.
• 36:32
  SPEAKER_02: GDPR gives European users actual rights.
• 36:35
  SPEAKER_02: California's AB 2013 gives transparency but not opt-out enforcement.
• 36:40
  SPEAKER_02: And for Atlassian users on Standard or Premium,
• 36:43
  SPEAKER_02: there is no opt-out at all for metadata collection.
• 36:46
  SPEAKER_02: A real choice requires three things,
• 36:48
  SPEAKER_02: awareness that the choice exists,
• 36:50
  SPEAKER_02: understanding of what each option means,
• 36:53
  SPEAKER_02: and confidence that your choice will be respected.
• 36:56
  SPEAKER_02: Most platforms give you the first if you're technically curious,
• 36:59
  SPEAKER_02: the second if you read the fine print,
• 37:02
  SPEAKER_02: the third,
• 37:03
  SPEAKER_02: you're trusting the same company that changed its terms without your input to honor
• 37:07
  SPEAKER_02: the toggle it gave you.
• 37:09
  SPEAKER_02: I think your listeners should decide for themselves whether that is a real choice
• 37:13
  SPEAKER_02: or just a gesture.
• 37:15
  SPEAKER_01: Yeah, that's a great point.
• 37:16
  SPEAKER_01: About two years ago,
• 37:18
  SPEAKER_01: Meadow announced that they were going to start using all of our pictures
• 37:21
  SPEAKER_01: and early information.
• 37:23
  SPEAKER_01: And, you know, some of us weren't even able to opt out.
• 37:26
  SPEAKER_01: They had this opt out mechanism that I put in mind and they said,
• 37:29
  SPEAKER_01: no, we don't have to honor that.
• 37:30
  SPEAKER_01: So we're not going to say, okay, then I'm going to delete everything.
• 37:34
  SPEAKER_01: And I did, I just deleted years of pictures and texts and everything.
• 37:39
  SPEAKER_01: But afterwards I was thinking I have almost no confidence that they aren't going
• 37:44
  SPEAKER_01: to use it anyway.
• 37:45
  SPEAKER_01: It's on backups.
• 37:46
  SPEAKER_01: They already had it.
• 37:48
  SPEAKER_01: I wouldn't want to count on it not being used.
• 37:50
  SPEAKER_01: Some of my guests have said they like what they call a billboard rule now,
• 37:53
  SPEAKER_01: where if they want to see it on a billboard along the highway,
• 37:56
  SPEAKER_01: they don't want to share it in any of those systems because they don't trust
• 38:00
  SPEAKER_01: the companies to not use it if they say they won't.
• 38:05
  SPEAKER_03: When cloud was booming,
• 38:07
  SPEAKER_03: there was this saying going around, if anything is cloud once, it's there forever.
• 38:13
  SPEAKER_03: So what does that even mean?
• 38:17
  SPEAKER_01: So has a company's use of your personal data
• 38:20
  SPEAKER_01: and content ever created any specific issues for you,
• 38:23
  SPEAKER_01: like privacy violations or phishing attacks or any loss of income,
• 38:28
  SPEAKER_01: any examples like that?
• 38:30
  SPEAKER_02: Yes, and the one that made me the most angry has nothing to do with AI companies directly.
• 38:36
  SPEAKER_02: It's about how data flows when it leaves your hands, regardless of what you consented to.
• 38:42
  SPEAKER_02: In mid 2024, I ran a credit inquiry to explore buying a house.
• 38:47
  SPEAKER_02: Standard process, I opted out of third-party data sharing completely.
• 38:51
  SPEAKER_02: I checked the boxes, did what you were supposed to do.
• 38:54
  SPEAKER_02: However, within days, my phone became unusable.
• 38:58
  SPEAKER_02: Robo calls, marketing calls, scam calls, all related to mortgage lending offers.
• 39:04
  SPEAKER_02: Not one or two, a constant stream that has not stopped to this day.
• 39:09
  SPEAKER_02: Two years later, I still get them.
• 39:12
  SPEAKER_02: It's frustrating.
• 39:13
  SPEAKER_02: What happened is something called trigger leads.
• 39:16
  SPEAKER_02: When a lender pulls your credit,
• 39:17
  SPEAKER_02: the credit bureaus,
• 39:18
  SPEAKER_02: Equifax,
• 39:19
  SPEAKER_02: Experian,
• 39:20
  SPEAKER_02: TransUnion can legally sell that inquiry to other lenders and marketers,
• 39:25
  SPEAKER_02: not your credit reports, just the fact that you applied.
• 39:28
  SPEAKER_02: But that fact alone is enough to generate a lead, and that lead gets sold and resold.
• 39:34
  SPEAKER_03: The CFPB received 5 million complaints about credit reporting agencies and lenders
• 39:40
  SPEAKER_03: in 2025 alone.
• 39:43
  SPEAKER_03: A 3,700% increase from 2019.
• 39:46
  SPEAKER_03: The problem got so bad that Congress passed the Home Buyers Privacy Protection Act,
• 39:51
  SPEAKER_03: signed in September last year,
• 39:53
  SPEAKER_03: which bans trigger leads starting March 2026, except in narrow circumstances.
• 39:59
  SPEAKER_03: Congress literally had to write a law because opting out did not work.
• 40:04
  SPEAKER_03: And here's the part that connects to everything else we have been talking about.
• 40:07
  SPEAKER_03: I opted out, I followed the process, but it did not matter.
• 40:11
  SPEAKER_03: The system was designed so that my credit inquiry was a product before I finished
• 40:15
  SPEAKER_03: the application.
• 40:16
  SPEAKER_03: That is the same structural problem as AI training data consent.
• 40:20
  SPEAKER_03: You can check every opt out box on LinkedIn or GitHub or Atlassian,
• 40:25
  SPEAKER_03: but the data that has already moved has already moved.
• 40:29
  SPEAKER_03: The architecture is built to extract first and process objections later.
• 40:35
  SPEAKER_03: My phone ringing
• 40:36
  SPEAKER_02: at 9 p.m.
• 40:37
  SPEAKER_02: with a fake landing pro offer is the most mundane,
• 40:40
  SPEAKER_02: most annoying,
• 40:41
  SPEAKER_02: most daily version of the same constant failure that,
• 40:44
  SPEAKER_02: Karen, you have been documenting in the AI training data context.
• 40:48
  SPEAKER_02: It is just less interesting to write about because it is not a frontier model.
• 40:52
  SPEAKER_02: It's a spreadsheet of phone numbers sold for three cents each probably.
• 40:56
  SPEAKER_02: And on the professional side,
• 40:58
  SPEAKER_02: the issue compounds because I need to govern data access in
• 41:01
  SPEAKER_02: a further regulated environment.
• 41:03
  SPEAKER_02: So when companies like Atlassian just announced abruptly
• 41:08
  SPEAKER_02: that they would use product interaction data to train their models
• 41:12
  SPEAKER_02: and metadata collection would be mandatory,
• 41:15
  SPEAKER_02: I had to stop my actual work and draft a risk assessment.
• 41:19
  SPEAKER_02: Our JIRA and Confluence instances contains print planning,
• 41:22
  SPEAKER_02: incident descriptions,
• 41:24
  SPEAKER_02: architectural docs, discussion threads referencing student-facing systems.
• 41:28
  SPEAKER_02: This is not abstract.
• 41:29
  SPEAKER_02: If that trains a model and something surfaces that traces to a student record,
• 41:34
  SPEAKER_02: I'm the person explaining it to the compliance office.
• 41:37
  SPEAKER_02: That is the dual cost.
• 41:39
  SPEAKER_02: Personally, my phone is a liability because I ran a credit check two years ago.
• 41:43
  SPEAKER_02: Professionally,
• 41:44
  SPEAKER_02: I'm not spending real hours on vendor data policy review that did not exist
• 41:49
  SPEAKER_02: in my job description.
• 41:51
  SPEAKER_02: And in both cases, I did the right thing.
• 41:53
  SPEAKER_02: I opted out.
• 41:54
  SPEAKER_02: I read the terms.
• 41:56
  SPEAKER_02: It just did not matter.
• 41:58
  SPEAKER_02: So for your listeners,
• 41:59
  SPEAKER_02: if you have ever opted out of something and then watched the thing you opted out
• 42:03
  SPEAKER_02: of happen anyway, that's not a bug.
• 42:06
  SPEAKER_02: That's just the business model.
• 42:08
  SPEAKER_02: And it's the same business model whether the product is a mortgage lead
• 42:12
  SPEAKER_02: an AI training dataset.
• 42:14
  SPEAKER_01: Yeah, and one thing that I think is worth noting is that this use of our data,
• 42:19
  SPEAKER_01: it's not really new with the generative AI tools that just really hit broad usage
• 42:25
  SPEAKER_01: a few years ago.
• 42:25
  SPEAKER_01: They've been doing that really for decades.
• 42:28
  SPEAKER_01: What makes it more insidious is that once our data is out there and it's been sold,
• 42:33
  SPEAKER_01: once it gets pulled into an AI system as training data,
• 42:36
  SPEAKER_01: it's never going to come back out.
• 42:39
  SPEAKER_01: So whatever mistakes or connections between a phone number and some other data
• 42:43
  SPEAKER_01: about you, there's just no way it's ever going to get corrected.
• 42:47
  SPEAKER_01: And that was already an existing problem, but it just amplifies it and makes it more serious.
• 42:53
  SPEAKER_02: Yeah.
• 42:53
  SPEAKER_02: I mean, previously we could still trace it back to where the data leaked.
• 42:58
  SPEAKER_02: But now we cannot even know when and where the data leak or data sharing happened.
• 43:06
  SPEAKER_01: Yeah.
• 43:06
  SPEAKER_01: So last question, we've talked a little bit about trust.
• 43:09
  SPEAKER_01: We see public distrust of AI and tech companies has been growing and
• 43:13
  SPEAKER_01: for some good reasons, I think.
• 43:15
  SPEAKER_01: I'm wondering what you think is the most important thing
• 43:18
  SPEAKER_01: that these companies would need to do to earn and to keep your trust,
• 43:22
  SPEAKER_01: if that's even possible.
• 43:24
  SPEAKER_01: Some folks I've talked to say they don't feel they could ever really trust them.
• 43:27
  SPEAKER_01: So I'm wondering what your thoughts are.
• 43:29
  SPEAKER_02: Yeah, it's really going to be very hard to trust again, like 100%.
• 43:34
  SPEAKER_02: But at least I think the one thing I would really like to see is default to off, you know?
• 43:40
  SPEAKER_02: And I know I have been talking about it in previous questions too,
• 43:45
  SPEAKER_02: but I think this is the single most important thing for me.
• 43:49
  SPEAKER_02: I know it sounds simple, but I mean it to sound simple.
• 43:53
  SPEAKER_02: The complexity is a distraction.
• 43:55
  SPEAKER_02: Every AI company that wants my trust should adopt one principle.
• 43:59
  SPEAKER_02: No user's data or content is used for training unless
• 44:03
  SPEAKER_02: the user explicitly affirmatively opts in not opt out opt in not buried in settings
• 44:10
  SPEAKER_02: on the sign up screen before the account is created and
• 44:14
  SPEAKER_02: the reason I'm picking this one over transparency over compensation
• 44:18
  SPEAKER_02: over ethic boards is because it is the one that cannot be performed You can write
• 44:25
  SPEAKER_02: a beautiful AI ethics statement and ignore it operationally.
• 44:29
  SPEAKER_02: I've seen organizations do that.
• 44:30
  SPEAKER_02: You can publish a model card and leave out the uncomfortable parts.
• 44:34
  SPEAKER_02: You can create an ethics board and then overwrite its recommendations.
• 44:38
  SPEAKER_02: But default to off is binary.
• 44:40
  SPEAKER_02: Either the toggle starts off or it just doesn't.
• 44:44
  SPEAKER_02: Either the user opted in before the data was used or they did not.
• 44:49
  SPEAKER_02: There is no way to
• 44:49
  SPEAKER_03: fake it.
• 44:51
  SPEAKER_03: And specific implementation ideas, I think I can give a few.
• 44:55
  SPEAKER_03: First, data provenance labeling.
• 44:57
  SPEAKER_03: California's AB 2013 that took effect now requires AI developers to use
• 45:02
  SPEAKER_03: and disclose what data trained their models,
• 45:05
  SPEAKER_03: including whether copyrighted materials or personal information was included.
• 45:10
  SPEAKER_03: That's a start, but it is just one state and it is disclosure without enforcement teeth.
• 45:16
  SPEAKER_02: And the EU Act enforcement provisions will require training data transparency
• 45:21
  SPEAKER_02: for general purpose AI models with actual fines behind it up to 15 million euros
• 45:26
  SPEAKER_02: or 3% of global revenue.
• 45:28
  SPEAKER_02: We need the California standard as a federal baseline with the EU's enforcement model behind it.
• 45:35
  SPEAKER_02: Disclosure without consequences is just a press release.
• 45:39
  SPEAKER_02: Secondly, I think retroactive constant mechanism is very important.
• 45:44
  SPEAKER_02: Every platform I have discussed today,
• 45:47
  SPEAKER_02: LinkedIn, GitHub, Atlassian, X, says the opt-out only applies going forward.
• 45:52
  SPEAKER_02: Everything already ingested is already used.
• 45:56
  SPEAKER_02: That cannot be the final answer.
• 45:57
  SPEAKER_02: If you can ingest my data without asking,
• 46:00
  SPEAKER_02: you can build the infrastructure to remove it when I ask.
• 46:03
  SPEAKER_02: Atlasian says they will retain models within 90 days of an opt-out
• 46:08
  SPEAKER_02: and remove in-app data within 30 days of deletion.
• 46:11
  SPEAKER_02: That is actually a good model.
• 46:13
  SPEAKER_02: Other companies should match it.
• 46:15
  SPEAKER_02: Most will never do.
• 46:17
  SPEAKER_02: And lastly, stop getting opt-out behind premium tiers, please.
• 46:22
  SPEAKER_02: This is really crucial.
• 46:25
  SPEAKER_02: Why should I need to pay more and more just to be able to have the opt-out option?
• 46:32
  SPEAKER_02: If it becomes the industry norm, privacy will become a luxury feature.
• 46:38
  SPEAKER_02: The people with the least institutional power,
• 46:40
  SPEAKER_02: the freelancers,
• 46:41
  SPEAKER_02: the small teams, the individual developers will get the least production.
• 46:45
  SPEAKER_02: That is the opposite of how it should work.
• 46:49
  SPEAKER_02: So the foundation is needed to be defaulted off.
• 46:53
  SPEAKER_02: Everything else is kind of negotiable for me if I can just get this foundation there.
• 47:00
  SPEAKER_01: That makes good sense.
• 47:01
  SPEAKER_01: Yeah, you have to start from the point where if you have the default off,
• 47:06
  SPEAKER_01: then hopefully they would be transparent about any opting in that would
• 47:11
  SPEAKER_00: be happening
• 47:11
  SPEAKER_00: and telling you what you're opting into and things like that.
• 47:14
  SPEAKER_00: But yeah, you have to have a starting point.
• 47:15
  SPEAKER_01: I want to thank you, Samaya, for sharing your time with me for this interview.
• 47:18
  SPEAKER_01: Is there anything else that you would like to share with our audience?
• 47:21
  SPEAKER_02: I just want to talk to the person listening right now who is exhausted,
• 47:26
  SPEAKER_02: not exhausted from work,
• 47:28
  SPEAKER_02: exhausted from the feeling that AI means you should be doing more,
• 47:31
  SPEAKER_02: that everyone else figured out everything about it and you are so behind.
• 47:37
  SPEAKER_02: If you're not automating something, you are failing.
• 47:39
  SPEAKER_02: This is completely wrong.
• 47:41
  SPEAKER_02: I want to tell you something about why I built all these systems,
• 47:44
  SPEAKER_02: because the reason is not what you might think.
• 47:47
  SPEAKER_02: I'm an eldest daughter in a Bangladeshi family.
• 47:50
  SPEAKER_02: I came to the U.S.
• 47:52
  SPEAKER_02: The wearing I grew up with is anticipate everything,
• 47:55
  SPEAKER_02: carry the load, never drop the ball, never be the reason something fails.
• 47:59
  SPEAKER_02: If you grew up in this way, you know it's not a personality trait.
• 48:03
  SPEAKER_02: It's an operating system.
• 48:05
  SPEAKER_02: It runs in the background of every decision.
• 48:08
  SPEAKER_02: Now give that person AI.
• 48:10
  SPEAKER_02: It has me infinite capacity to do exactly what I was already worried to do.
• 48:15
  SPEAKER_02: Before AI, I could carry a lot.
• 48:17
  SPEAKER_02: Now I can carry almost everything.
• 48:19
  SPEAKER_02: I can automate my newsletter, my job search, my content calendar.
• 48:23
  SPEAKER_02: I can do all of it.
• 48:25
  SPEAKER_02: So I do all of it.
• 48:26
  SPEAKER_02: And the result is that I produce more than I ever have.
• 48:29
  SPEAKER_02: So I'm more anxious than I have ever been because the finish line just moved.
• 48:35
  SPEAKER_02: When I could to just X, doing X was enough.
• 48:39
  SPEAKER_02: But now I know that I can do five times.
• 48:42
  SPEAKER_02: So if I ever do four times, it just feels like I'm falling behind.
• 48:47
  SPEAKER_02: In April,
• 48:48
  SPEAKER_02: I caught myself at 1 AM building a new agent to automate something I could have done
• 48:53
  SPEAKER_02: in 20 minutes by hand.
• 48:56
  SPEAKER_02: That's the moment I stopped because it was a bad idea.
• 48:59
  SPEAKER_02: I realized I was not building it to save time.
• 49:02
  SPEAKER_02: I was building it because finishing felt like permission to rest.
• 49:06
  SPEAKER_02: I was not going to give myself permission anyway.
• 49:08
  SPEAKER_02: So I closed the laptop.
• 49:10
  SPEAKER_02: That was the first time in months.
• 49:12
  SPEAKER_02: And the moment I realized it's becoming unhealthy.
• 49:15
  SPEAKER_02: So now I have two rules on myself,
• 49:18
  SPEAKER_02: no building after 10 p.m. And one day a week, I will not open any agentic systems.
• 49:24
  SPEAKER_02: Obviously I break those rules sometimes, but at least the rule exists, right?
• 49:28
  SPEAKER_02: So it's different from before.
• 49:30
  SPEAKER_02: I think your listeners feel the version of it.
• 49:33
  SPEAKER_02: Applies to them too.
• 49:34
  SPEAKER_02: Maybe they're not eldest daughters or immigrants,
• 49:36
  SPEAKER_02: but if you have ever looked at all the things they could help you do
• 49:40
  SPEAKER_02: and felt not
• 49:41
  SPEAKER_03: excited but overwhelmed,
• 49:43
  SPEAKER_03: that's the same worrying under a different name.
• 49:45
  SPEAKER_03: Rescaling capacity with our scaling piece,
• 49:48
  SPEAKER_03: and I worry we are about to do that to millions of people and call it productivity.
• 49:53
  SPEAKER_03: So if you're listening and you feel like you cannot keep up, you're not behind.
• 49:57
  SPEAKER_03: The finish line just moved.
• 49:59
  SPEAKER_03: It was not your fault and the bravest thing you can do with the tool
• 50:03
  SPEAKER_03: that offers infinite capacity is choose not to use all of it.
• 50:08
  SPEAKER_03: And I obviously do not have it figured out.
• 50:10
  SPEAKER_03: I still check my agents before breakfast,
• 50:12
  SPEAKER_03: but I'm learning that the point of building a system is that you can do everything
• 50:16
  SPEAKER_03: to not choose to let it do everything.
• 50:21
  SPEAKER_02: Thank you so much, Karen, for giving me the space to say that out loud.
• 50:25
  SPEAKER_01: Thank you.
• 50:26
  SPEAKER_01: And do you want to let people know where they can find you online
• 50:29
  SPEAKER_01: if they're interested in reading more of your writing?
• 50:32
  SPEAKER_02: Yes, absolutely.
• 50:33
  SPEAKER_02: I'm almost 24-7 available on LinkedIn.
• 50:38
  SPEAKER_02: I have a weekly newsletter in Substack.
• 50:41
  SPEAKER_02: You can feel free to subscribe to it if you want to learn more
• 50:44
  SPEAKER_02: about Enterprise AI Adoption from a practitioner's site.
• 50:47
  SPEAKER_02: I'm also making educational content in Instagram almost daily.
• 50:52
  SPEAKER_02: You can find me there as well.
• 50:55
  SPEAKER_01: We'll have a LinkedIn link in the article, but I know some people listen instead of reading.
• 50:58
  SPEAKER_02: LinkedIn, I don't even know if they type my name, it even comes.
• 51:03
  SPEAKER_02: When they type Sumayashwini, it goes to another person in LinkedIn somehow.
• 51:09
  SPEAKER_02: That's the only problem.
• 51:12
  SPEAKER_02: Exactly, it goes to another person by default because they have more followers.
• 51:17
  SPEAKER_01: So maybe the best way for someone to find you is to find you on Substack
• 51:20
  SPEAKER_01: and connect there.
• 51:21
  SPEAKER_01: Say the name of your newsletter.
• 51:23
  SPEAKER_01: My newsletter name is GroundTruth.
• 51:26
  SPEAKER_01: All right.
• 51:27
  SPEAKER_01: Well, we will include the links in the article.
• 51:28
  SPEAKER_01: So if anyone doesn't get it from the recording, they'll be able to pick it up from there.
• 51:33
  SPEAKER_01: I really enjoyed talking with you and thank you so much for making time for the interview.
• 51:37
  SPEAKER_02: Thank you for giving me such a long time, Karen.
• 51:41
  SPEAKER_00: About this interview series and newsletter.
• 51:44
  SPEAKER_00: This post is part of our AI 6P interview series on AI software and wetware.
• 51:51
  SPEAKER_00: It showcases how real people around the world are using their wetware brains
• 51:56
  SPEAKER_00: and human intelligence with AI-based software tools who are being affected by AI.
• 52:03
  SPEAKER_00: And we're all being affected by AI nowadays in our daily lives, perhaps more than we realize.
• 52:09
  SPEAKER_00: For some examples, see post, but I don't use AI.
• 52:14
  SPEAKER_00: We want to hear from a diverse pool of people worldwide in a variety of roles.
• 52:19
  SPEAKER_00: If you're interested in being a featured interview guest,
• 52:22
  SPEAKER_00: anonymous or with credit, please get in touch.
• 52:29
  SPEAKER_00: All new posts are free to read and listen to.
• 52:38
  SPEAKER_00: To automatically receive new AI6P posts and support our work, consider becoming a subscriber.
• 52:44
  SPEAKER_00: It's free.