Latest / The Effective Data Scientist / Using Frameworks for Prompts for Better Outcomes

Transcript

Using Frameworks for Prompts for Better Outcomes · The Effective Data Scientist

• 0:07
  Speaker 1: Welcome to the Effective Data Scientist Podcast.
• 0:20
  Speaker 1: This podcast is designed to help include your skills, save focus manage successful projects and have fun at work.
• 0:28
  Speaker 1: Be part of our community.
• 0:30
  Speaker 0: this
• 0:31
  Speaker 1: is an independent podcast and the views expressed are all over.
• 0:43
  Speaker 0: Hello everyone!
• 0:44
  Speaker 0: And welcome back to the effective data scientist.
• 0:48
  Speaker 0: Hi Paolo how I drink?
• 0:50
  Speaker 1: I have this.
• 0:51
  Speaker 1: uh I'm doing well.
• 0:52
  Speaker 1: what about you?
• 0:53
  Speaker 0: I'm doing fine, hoping for the sun to come back.
• 0:57
  Speaker 1: Same here it seems that winter is back once again but let's hope an improved weather in coming days.
• 1:06
  Speaker 0: Stay positive!
• 1:08
  Speaker 0: So i am super excited today Paolo because you and me will be discussing prompting.
• 1:14
  Speaker 0: what really means why matters how frameworks can improve quality reliability of AI inputs.
• 1:23
  Speaker 0: So today's episode is about using frameworks for prompts, for better outcomes.
• 1:31
  Speaker 0: And let me start with a situation many of us have experienced.
• 1:35
  Speaker 0: maybe you too.
• 1:37
  Speaker 0: so ask an AI tool question and the answer sounds confident it well written yet you hesitate to actually use it.
• 1:47
  Speaker 0: does that sound familiar to you Paolo?
• 1:50
  Speaker 1: Yes family scenario, because sometimes it seems easy to get in front of the generative AI tool and askively.
• 2:00
  Speaker 1: And the outcome is not what you were expecting.
• 2:04
  Speaker 1: now Because although things are always changing In a sense that LLMs are improving I think its really important To apply The best principles and start very focused, and tailored to the question because it's difficult to get better answers refining an initial prompt which is not very well structured.
• 2:34
  Speaker 1: or It's better to start with a well-structured prompt instead of starting in simple and trying to refine later.
• 2:43
  Speaker 1: Which is possible but it's suboptimal.
• 2:46
  Speaker 1: I
• 2:46
  Speaker 0: agree with you.
• 2:48
  Speaker 0: So for our listeners who are not familiar with the word prompting, I assume there some of us that aren't familiar with terminology we're using.
• 2:58
  Speaker 0: But just going back one step... When you talk about prompting it simply means how to instruct an AI system what-to do?
• 3:08
  Speaker 0: A prompt is input given It's a question, task description and instruction.
• 3:15
  Speaker 0: And it's not just asking a question, It is much closer to writing a good analytical brief.
• 3:22
  Speaker 0: Defining assumptions and setting constraints being explicit about what correct means.
• 3:29
  Speaker 0: So I learned this myself as data scientist.
• 3:33
  Speaker 0: we already know that the less assumptions are confusing The more questions can generate.
• 3:40
  Speaker 0: Missing assumptions lead to wrong conclusions, for example.
• 3:44
  Speaker 0: Or a small misunderstanding can have very big consequences The more detailed you explain the background I think assumptions and constraints And what is exactly expected.
• 3:58
  Speaker 0: Is there way to go?
• 4:00
  Speaker 0: I would say
• 4:02
  Speaker 1: Yeah...I think you introduced me To the word of Brahms frameworks because I wasn't aware of different prompting frameworks that exist and they span from the easier to more structured.
• 4:20
  Speaker 1: Do you have some examples?
• 4:22
  Speaker 0: Yeah, sure!
• 4:23
  Speaker 0: I was not aware about it too... I just accidentally learned about prompting in one our learning sessions at my company And so I learned that the prompting framework is simply a structured way to reduce ambiguity by answering few key questions up from, for example.
• 4:42
  Speaker 0: What should they AI be?
• 4:44
  Speaker 0: what Is The goal and what is the context?
• 4:47
  Speaker 0: and what kind of output do i expect?
• 4:50
  Speaker 0: and how Should the AI reason or proceed?
• 4:53
  Speaker 0: So this was something I learned and apparently there are varieties Of examples of prompting frameworks That we can use And I was introduced to ROSE's framework, which stands for R-Role O-Objective S Scenario and E for Expected Solution and Steps.
• 5:17
  Speaker 0: This is one framework that we can use better explaining what do you expect?
• 5:23
  Speaker 0: To better formulate the problem statement?
• 5:30
  Speaker 0: This was one example that I learned from the learning sessions and i was curious about use.
• 5:36
  Speaker 0: And this is something, do you have any examples?
• 5:59
  Speaker 1: and you can use certain sources like data, literature guidelines.
• 6:06
  Speaker 1: And also set up the expectations in terms of the length or the plan T level of analytical details and some basic layers of assumptions .And then they You Can Also Set Up.
• 6:24
  Speaker 1: The Style In The Tone Like For Example Could Be less formal since we are writing a psychometric analysis plan and maybe statisticians in the pharmaceutical industry, I'm not familiar with the language.
• 6:38
  Speaker 1: And the science behind the analytical plan then may be.
• 6:41
  Speaker 1: you can include that in the prompt in terms of the styling expectations also to include more context for the methods being used on it improves a lot The first examples and the very first drafts.
• 6:59
  Speaker 1: And then it becomes easier to iterate quickly, get a solution faster.
• 7:07
  Speaker 1: Of course I'm currently only experimenting.
• 7:12
  Speaker 1: It's important because when i was learning these frameworks one thing is that you can use them also as an inspiration each framework for your specific case.
• 7:29
  Speaker 1: The narrative doesn't make sense to you?
• 7:32
  Speaker 0: I agree with you, there are varieties and then... For example this Roses case, the Roses Framework is it seems that its used when the task is complex When the context really matters And you want a structured reasoning.
• 7:47
  Speaker 0: So also mentioned about style like which I can correspond or I can resonate with another type of framework, which is known as CRISP stands for context role instruction style parameters.
• 8:03
  Speaker 0: And this is particularly useful when format and structure are critical.
• 8:08
  Speaker 0: so different prompting frameworks were different tasks.
• 8:11
  Speaker 0: As you mentioned it could be used depending on what your working and you can make a use of different prompting frameworks.
• 8:22
  Speaker 1: It's really nice to read about them, I was impressed by the number and quality of the acronyms.
• 8:31
  Speaker 1: You can start very simple for example with the APE framework which stands for action purpose expectation Which is very basic.
• 8:42
  Speaker 1: So what do we want?
• 8:44
  Speaker 1: to do, the purpose required.
• 8:46
  Speaker 1: The expectation is the formal style requirements and there are many examples I've found for example for creative work in marketing.
• 9:00
  Speaker 1: but you can also find examples in a science And You Can Also Reuse Some Of The Case Study You Find In Other Fields For Your Specific Career Expertise Your Area of Work.
• 9:14
  Speaker 1: Another thing I want to explore is the principle of front-end in general, because frameworks just have a first approach that can work for you.
• 9:26
  Speaker 1: But it's also important to follow development with large language models and understand how do they work?
• 9:35
  Speaker 1: And what was best way to interact with them.
• 9:39
  Speaker 0: That would be really great if we knew like learning your language a foreign language and really be able to understand what is meant, so it will help you communicate better.
• 9:51
  Speaker 0: Learning what's behind the program.
• 9:54
  Speaker 1: Starting from the framework... It's important make your own research and pick up from beginning the best framework that applies for work and fine-tune elements of the framework.
• 10:12
  Speaker 0: I think that would be great.
• 10:15
  Speaker 0: And while we were talking, I was switching to different ideas like what can ask you and the idea is thinking where could apply?
• 10:25
  Speaker 0: What are challenges facing at work or which kind of problems have in mind for this scenario?
• 10:33
  Speaker 0: maybe i can ask about your input on how do approach it The scenario when you have data, which is not in the right format.
• 10:44
  Speaker 0: Like with statistician we need that data in the correct format and do we have to have it in a tabular form?
• 10:51
  Speaker 0: And NOT IN THE EXCELFORM with merged cells... ...and A LOT OF INCONSISTENCY in the cells & labelling.
• 10:59
  Speaker 0: So which framework would you use for creating a prompt that will help you to transform this data into the format that you will be using.
• 11:13
  Speaker 0: So, it was something I was struggling and in my company... In my role i get data from many different formats.
• 11:21
  Speaker 0: It comes from PDF reports, Excel records with not proper formatting for statistical analysis because I was afraid that if it changes one data or value, this will have a drastic impact.
• 11:42
  Speaker 0: Because every data we generate is very expensive.
• 11:46
  Speaker 0: and how to make sure i get the correct values back from what I provided in the AI tool?
• 11:56
  Speaker 1: I think its important to discuss the ideal framework to stop.
• 12:03
  Speaker 1: For example, I think that trace framework or trace framework are great examples because in trace defect on the element is requirements which add the technical specifications for the final output.
• 12:19
  Speaker 1: but i think it's very important setup of your data and then the expectations for how they should look like at the end of the transformation.
• 12:40
  Speaker 1: For example, in general LLMs are now very capable of doing an at-the-end tabular data is a very simple example language.
• 12:53
  Speaker 1: so it's chain of symbols which have numbers with separators.
• 12:58
  Speaker 1: So if you are able to set up the requirements, the context and also the expected outputs in the best way then You can get a lot from your prompt.
• 13:12
  Speaker 1: There is an evolving field for example how to generate the prompt automatically From LLMMs To deal with specific situations.
• 13:23
  Speaker 1: This is quite frontier topic For now But for sure, we have to be careful with the technical requirements of this scenario and expectation.
• 13:34
  Speaker 0: Yeah thank you for the input that maybe I can share.
• 13:38
  Speaker 0: how i dealt in this scenario?
• 13:40
  Speaker 0: I did use AI itself for helping me create a better prompt!
• 13:47
  Speaker 0: So what I did is enroll...I took role And said You are senior data scientist With strong experience in pharmaceutical development.
• 13:57
  Speaker 0: Your objective is transform raw experimental data into an analysis ready tabular format without changing the underlying data.
• 14:08
  Speaker 0: And a critical constraints are do not modify infer impute correct or delete any data values, Do not assume units meanings relationships unless explicitly stated and do not rename variables unless explicitly unstructured.
• 14:27
  Speaker 0: If ambiguity exists, stop and ask clarification questions.
• 14:32
  Speaker 0: In the task I said describe how you interpret the current data structure a target tabular structure And explicitly list every transformation You plan to apply.
• 14:45
  Speaker 0: Wait for my confirmation before performing any transformation.
• 14:49
  Speaker 0: Last but not least there are some verification requirements and also style verification requirements, I said provide the before and after row and column counts.
• 15:02
  Speaker 0: And some more details in style a brief precise cautious an explicit and treat these as regulated development data.
• 15:11
  Speaker 0: so this is very important!
• 15:14
  Speaker 0: i think it was okay for me but should apply several times to see if get results that are promising.
• 15:24
  Speaker 1: Sometimes when I refine the prompt after a few iterations, I prefer to restart another chapter.
• 15:32
  Speaker 1: To clean up context because i see that After a few iteration things are not improving as expected
• 15:40
  Speaker 0: and you find yourself in forest of big data trying process all outcomes right?
• 15:48
  Speaker 1: Just in the middle of exploration, I mean it's a bit heuristic.
• 15:52
  Speaker 1: i don't have any data or source to support that.
• 15:55
  Speaker 1: It is also good to ask the A-Tool To provide you with an Aswerra Or once You Have Something Good With Your Front Then You Can Also Directly Ask The A-tool To Improve Directly The Front So you can make a better use of the GenAI tool also asking to improve your content.
• 16:22
  Speaker 0: Yeah, that's really great strategy.
• 16:24
  Speaker 0: and do you also use different AI tools?
• 16:28
  Speaker 0: Like I used the same prompt using different Ai tools and see if you get two different results.
• 16:35
  Speaker 1: I typically don't do it but just for practical reasons because usually working with my company laptop for example or my personal one more on frequency then we are a bit bounded in terms of the pulse you can use.
• 16:54
  Speaker 1: but, In general I tend to pre-one AI tool For specific task and i don't switch across tools for same tasks.
• 17:02
  Speaker 1: And...for example..in terms of coding I prefer work with cloud while for tasks like writing or you know searching and fine with chat gpt also.
• 17:16
  Speaker 1: so it depends more on the type of task.
• 17:20
  Speaker 1: but i'm not always curious about what could be the result if I ask different general tools.
• 17:28
  Speaker 0: yeah
• 17:29
  Speaker 1: well that's a good advice, to explore.
• 17:32
  Speaker 0: Yeah, I do share your limitations with respect to the work environment.
• 17:38
  Speaker 0: We are using our own validated AI tool for that and sometimes i really wish that i could just have more options To see if i get the same results Using different AI tools like the same question Just different AI Tools And See What Happens.
• 17:53
  Speaker 0: But I'd Share The Limitation That You Mentioned.
• 17:56
  Speaker 0: And yeah, so for general tasks maybe it would be a good idea to see like not work related task or general task.
• 18:05
  Speaker 0: It could be nice to see what are the outcomes that we get and if they differ... If one answer is better or not?
• 18:12
  Speaker 0: So this will be something maybe someone can try.
• 18:17
  Speaker 0: Yeah
• 18:17
  Speaker 1: I agree.
• 18:18
  Speaker 0: We talked about different frameworks for prompting.
• 18:23
  Speaker 0: Maybe we can recap.
• 18:25
  Speaker 0: We talked about roses,
• 18:27
  Speaker 1: we named a few like trace aid.
• 18:30
  Speaker 0: And there are other prompts too Like care context action result an example.
• 18:36
  Speaker 0: and yeah A lot of options one can use.
• 18:41
  Speaker 0: I think if you browse in internet You find a lot Of options and a lot of articles that Are available with some applications shown.
• 18:50
  Speaker 0: It's really nice to see how Use them in which kind of situations to use them and what would be your final take away or message?
• 19:02
  Speaker 0: To the listeners.
• 19:04
  Speaker 1: I think it's really important to explore The framework that applies best for your situation And then experiment and find tool for your day-to-day work.
• 19:18
  Speaker 1: Stay tuned with the update, be careful because they use new ways of prompting or updates for large language models are overwhelming at this day.
• 19:31
  Speaker 1: Be a bit cautious but some exploration.
• 19:34
  Speaker 1: you can pick up one of these frameworks improve it and start in better way with your prompting.
• 19:42
  Speaker 0: I agree with you.
• 19:42
  Speaker 0: thank you Paolo for this fantastic discussion.
• 19:46
  Speaker 0: I think the general takeaway would be from my side, don't ask AI for answers.
• 19:52
  Speaker 0: Design how it should think better description and explaining what exactly do you expect that could be useful to get what you want?
• 20:02
  Speaker 0: So thank you Paolo!
• 20:05
  Speaker 0: It was a good start into this topic.
• 20:08
  Speaker 0: Let's continue with different topics regarding AI and data science in future episodes and thank you for the listeners, for tuning into the effective beta science.