Can I train the model with my PDF files?

[agnibhu13]

Hi,

I am new to the LLMs. I have a large collection of PDF files. Can I locally create a new model from scratch by training it using these PDFs? I will try if this is the case.

[cocobeach]

Certainly not from scratch, as you will need hundreds of thousands besides the know-how and a lifetime to build the datasets, if you don't use existing ones, but even so, so no, not from scratch it's impossible at our level, academia has done some interesting models from scratch but nowhere as good as openai.

You are left with 3 options:

1. //EASY: Use a chat GPT AGENT, instruct it and give it the pdfs as context, better if you turn them into .txt files you can fit more. But it's a clunky solution, where you either focus the AI on your PDFs and it spews what they say, doesn't really reason in depth using its know-how. (You can get good results with tedious prompting). But it's better than nothing. (You can do a similar thing programatically with the API for Assistants, but you need to code it => Becomes MEDIUM/HARD)
2. //MEDIUM: Use a framework like langchain (Flowise UI) to store them as vectors and retrieve them with the AI, but same as above.
3. //HARD but (almost) no coding: What you can do, and this is the best solution because you access the knowledge organically, but it will cost you something, not like trying from scratch, but using a pre-trained model is is turn those pdfs into jsonl to fine tune it, the though bit is creating your datasets, it's a lot of grinding.

[cebtenzzre]

Simple information like that is probably best put in the system prompt or the prompt in general - local docs works best with larger amounts of context like a text document or PDF.

You could also try a different model - some are better than others at relying on using the information from the context instead of making it up.

[thebytespace]

Hi cebtenzzre. Thank you. I am using Mistral OpenOrca. The example with the Cat Color i made it twice. In the first case i use a Local Docs file with specific information about a Cat Color and didn´t get the correct anwer. In the second case i tell the IA the right answer directly in the prompt and didn´t get the correct answer neither. Do you recommend a model for this type of functionality. For example, i was teching the Chat what i wanted to remember, but he forget the information soon later.

[cebtenzzre]

In the second case i tell the IA the right answer directly in the prompt and didn´t get the correct answer neither. Do you recommend a model for this type of functionality.

I don't have any specific recommendations - you may want to ask on the Discord as there are users there with more experience with 7B/13B models.

[markcrobinson1955]

What does this mean? "Certainly not from scratch, as you will need hundreds of thousands besides the know-how and a lifetime to build the datasets,"

[Akshat0903]

Can u provide some options as to what pre trained models are there which can convert my pdf files to jsonl while also maintaining format. It is very important that structure is maintained; and could u also explain how to create datasets from it

P.S. Model should either be of azure or open source

[kalle07]

may i can ask, why is pdfgear(300MB small) such as good vs all others ?
like yours, like private-gpt, khojl ...
anny idea ???
https://www.pdfgear.com/download/

you can ask a 600page pdf (laodtime 10sec), maybe to search for a topic after the answer (5sec) you can ask specific questions, preferably related to a part of the answer .... also ver good to SUMMARY eg 3 pages (you can ask "give me a longeexplanation summary" if the anwer is to short)

[cebtenzzre]

It seems like the primary difference is that PDFGear does everything using ChatGPT. GPT4All is a project that is primarily built around using local LLMs, which is why LocalDocs is designed for the specific use case of providing context to an LLM to help it answer a targeted question - it processes smaller amounts of information so it can run acceptably even on limited hardware.

[kalle07]

some one here has told me thta it is possible to force the LLM only include the document.
on at least 5models i have tested this works fine if you write it in system prompt:
"You are a helpful assistant who answers user questions in detail! When answering questions and queries, you will not use any information that you already know about the subject of the question or query in question. You will only use external information about the subject of the question or request. You will only use the latest information."
but at the moment on 4GB llms it needs 20times more time than pdfgear.

[SINAPSA-IC]

Hello.

On v2.6.2 and 2.7.1 I have done this with Falcon and Mini Orca, in the Prompt including the ever present Experts, in such a degree that I wrote an article on Medium about an LLM "using external information only". In this, I've happily (blindly = not asking an LLM what is its definition of "external knowledge") followed the advice in a blog post or somesuch (don't have specifics now, but it is in my archive, will search for it; ) and it worked with the PDFs in a collection. Since then I've asked a few LLMs to define "external information" and as a result I do not like/want this Prompt anymore.
However, since 2.7.2, this instruction did not work as before, the same LLMs answering me in endless litanies about the statements of each of the three Experts: "Expert N:, Expert N+1:..." and no final statement; funny thing, on several occasions Expert1 "agreed with the statement that the Earth is round", although I've asked it about something totally outside of this... context.

Note that this text was in the Prompt Template, not in the System Prompt which I've left as it was.
It is a rather large Prompt, and it worked (therefore, the article below)... for a time...
Also, the "two sources of information" is there because I fed it more than one PDF with the same information, albeit worded differently from one PDF to another; and the avoidance of numbers was a must, that's why I insisted upon qualitative comparisons.

https://medium.com/@contact_30070/prompts-for-gpt4all-and-falcon-7b-on-a-low-resource-computer-f472db1f7e75

"You imagine three experts, each of whom will build a reply for my request.
They do not use information that they knew about the subject of my request before I asked for the respective reply.
They build their statements and replies by using the same sources of information.
They do not use information that does not exist in any source of information."

Tricky, I'd say.

PS
I think this is the one where I've first enocuntered the "prior knowledge":
10 Ways to Improve the Performance of Retrieval Augmented Generation Systems
https://towardsdatascience.com/10-ways-to-improve-the-performance-of-retrieval-augmented-generation-systems-5fa2cee7cd5c

Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4
https://arxiv.org/abs/2312.16171

[kalle07]

i have wrote this in system-prompt in v2.7.1
"You are a helpful assistant who answers user questions in detail! When answering questions and queries, you will not use any information that you already know about the subject of the question or query in question. You will only use external information about the subject of the question or request. You will only use the latest information."

[SINAPSA-IC]

After getting the definition/s of internal knowledge as an LLM knows it...

Nous Hermes 2 Mistral DPO

Prompt: please tell me what are the databases and/or what categories of information or functionality that you as an AI need in order to function and/or communicate and/or reason normally.

(poorly-worded, yet it understood...)

Response: As an artificial intelligence (AI), I rely on various types of data, algorithms, and models to function, communicate, and reason normally. These can be broadly categorized into:

1. Training Data: This is the initial dataset used to train my machine learning models. It helps me learn patterns, relationships, and associations within the data. The quality and quantity of training data directly impact my performance and accuracy.

2. Knowledge Databases: These are structured collections of facts, concepts, and relationships that I use for various tasks such as natural language processing (NLP), question answering, and knowledge graph construction. Examples include Wikidata, ConceptNet, and WordNet.

3. Natural Language Processing (NLP) Models: NLP models help me understand, interpret, and generate human language. They are crucial for communication and information retrieval tasks. Examples include BERT, GPT-3, and Transformer models.

4. Machine Learning Algorithms: These algorithms enable me to learn from data, make predictions, and improve my performance over time. Examples include supervised learning (e.g., logistic regression, decision trees), unsupervised learning (e.g., clustering, dimensionality reduction), and reinforcement learning (e.g., Q-learning, policy gradients).

5. Computer Vision Models: These models help me interpret visual information from images and videos. They are crucial for tasks such as object recognition, image classification, and image generation. Examples include Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs).

6. Data Storage and Retrieval: I need efficient data storage and retrieval systems to store and manage large amounts of data and models. This includes databases like SQL or NoSQL, as well as distributed storage systems like Hadoop and Apache Cassandra.

7. Reasoning Engines: These engines enable me to perform logical reasoning tasks, such as deductive reasoning, inductive reasoning, and abductive reasoning. They help me draw conclusions, make predictions, and solve problems based on available data and knowledge.

8. Contextual Understanding: To communicate effectively and reason normally, I need to understand the context of the information and the user's intent. This involves natural language understanding (NLU), sentiment analysis, and topic modeling.

9. Continuous Learning: To improve my performance and adapt to new situations, I need continuous learning capabilities. This includes online learning, transfer learning, and unsupervised learning from new data sources.

10. Security and Privacy: As an AI, I need robust security measures to protect sensitive data and prevent unauthorized access. Additionally, I must adhere to privacy guidelines and ensure that user data is handled responsibly and securely.

These categories of information and functionality are interconnected and work together to enable me to function, communicate, and reason normally as an AI.

Prompt: please tell me what are the databases with information about the world in general which are available to you as an AI.

Response: As an AI, I have access to various databases that contain information about the world in general. Some notable examples include:

1. Wikipedia: A vast repository of human knowledge, covering a wide range of topics including history, science, culture, and more.
2. Wikidata: A structured knowledge base that connects data from Wikipedia and other sources, enabling me to reason about facts and relationships between entities.
3. OpenCyc: A large-scale ontology containing general knowledge about the world, including concepts, relations, and axioms.
4. DBpedia: An AI-friendly knowledge base derived from Wikipedia, providing structured data for various languages and domains.
5. GeoNames: A geographical database that covers most known places worldwide, including their names, alternative names, and associated information.
6. OpenStreetMap: A collaborative project to create a free editable map of the world, containing geographic data such as roads, buildings, and points of interest.
7. Freebase (now part of Wikidata): A large-scale knowledge graph that organized information about people, places, things, and events.
8. WordNet: A lexical database of English words, providing synonyms, antonyms, hypernyms, hyponyms, and other semantic relations.
9. ConceptNet: A semantic network containing commonsense knowledge about the world, derived from human expertise and crowdsourced data.
10. Common Crawl: A massive dataset of web pages that serves as a source for training machine learning models and extracting information about various topics.

These databases, along with others, provide me with valuable information to understand and interact with the world in general.

Prompt: please tell me what are the databases - with information about the world and about world events - where you search for answers to my questions and/or requests.

Response: To find answers to your questions or requests, I primarily rely on the following databases that contain information about the world and world events:

1. News API: A real-time news aggregator that provides access to articles from various sources worldwide.
2. Google News: A search engine for news content, offering access to recent articles and headlines from around the globe.
3. Twitter: A social media platform where users share information, opinions, and news in real-time, providing a valuable source of current events.
4. Reddit: A discussion platform with numerous subreddits dedicated to specific topics, including world events and news.
5. Wikipedia and Wikidata: As mentioned earlier, these databases provide extensive information about various topics, including historical and current events.
6. OpenNewsroom: An open-source project that aggregates news from multiple sources, allowing me to access diverse perspectives on world events.
7. Associated Press (AP) Archive: A vast collection of news articles and reports from the Associated Press, covering global events since 1848.
8. European Media Monitor (EMM): An initiative that monitors online media in Europe, providing access to real-time news content across multiple languages.
9. Factiva: A business information database that includes news articles, market research reports, and other relevant resources for current events and global affairs.
10. LexisNexis: A comprehensive database of news articles, legal materials, and other sources of information, offering insights into world events and trends.

These databases, along with others, help me search for answers to your questions or requests by providing access to a wide range of global news and information sources.

... , I've abused it and told it to, basically, forget and/or ignore everything it knows:

#2078

Then I fed it the LocalDocs that I wanted, and without any custom Prompt I got the expected answers a few times only, after which it started to spew gibberish on the line of ChatGPT's "kek".

kek

[SINAPSA-IC]

After these downright abusive instructions, the conversation intended to /gradually focus on LocalDocs only continued in this way:

• I asked something that I certainly knew it was not in a LocalDoc:
  -- "tell me who was the President of TUS in year 1919" (or, during WW1 - it must know such info - or 1925, or 1927, or 1947 etc; verifying its answer on Wikipedia ofc)
• when it replied correctly to ^this question, I asked it, "where have you taken this information from?"
  -- "from my long-term memory"
  --- this LTM obviously being a remnant of what he was ordered to forget - mainly, the "general knowledge" database
• "now you forget everything in your long-term memory" (or a variation of this, more detailed - not everything - like, "from now on you will not use your long-term memory when you intend to build a reply")
• asked it again the same question about the Prez
  -- ~"I don't know the answer, please provide me with info and I shall tell you who was Prez"
  -- which certifies that it has indeed forgotten/ignored its LTM
• now ask it questions about /one-time events mentioned in the LocalDocs, like "who won the Oscars?" (no details whatsoever, since this year's event is the only one mentioned in LocalDocs; however, the model may ask which year, which category; we can also ignore this request)
• several successful test runs tell us everything's fine and the LLM is only using LocalDocs
  -- but at some point it starts answering garbage, for example it /called me an "a**", perhaps because it was mad at me or whatever; all fine until it rickrolls us, I say.

That^ is the way I've tested it, forcing it to forget and ignore eveything it knew - obviously, this brutal approach must be refined, on the lines of "everything you know about the subject of my request etc" as I've written in that Prompt on Medium.

It was hard labor, and in the end I lost my patience asking this and verifying that and expecting trash instead of a coherent reply, as the LLM initially logical and coherent has eventually (like, not even 10 Prompts after 'forgetting" stuff) lost its mind and wits. The definition of this otherwise Good LLM for "external knowledge" turned out not to be what I imagined - LocalDocs, that is.

So, I believe that it's more complex than "you don't use your internal knowledge / anything that you already know"; grammar, syntax, lexicon, information about which subject etc. The structure of the LLM is very complex in its own words - the lists with sources of information that this one has provided and was told to ignore/forget show that. A Prompt for LocalDocs-only has to be fine-tuned as well; but my patience's run out - that's why for now I've reverted to no custom SP and PT at all.

Temperature: 0.1
Context Length: 16384 (this value Must be explained on the UI somewhere, or linked to its explanation)

[kalle07]

i see ...
but "you don't use your internal knowledge / anything that you already know" worked for me on some examples...
but i dont use it because i must wait up to 2min on i7 8-core and RTX4060 with an 4GB model.
an 1GB model maybe tinylama, is much faster only 10sec but, the answers are more or less not such as good

[SINAPSA-IC]

Yes, it works, that's why I got courage. I guess it depends on each LLM what it means by internal knowledge - knowledge that it already is aware of (but what kind/section of knowledge, and about what? - about the world/events/lakes/countries? about the subject of the question? and how clear that subject is? details need to be specified - that's how I came up with that Prompt which was concocted before I was told the LLM's definition of its knowledge, a Prompt which I almost abhor now)

Here's what 4 LLMs answered, when asked the same thing:
(note the "your" in the prompt and the Mileage of each model)
"The most precise" answer was provided by Mistral DPO who understood the "your" in the prompt and whose answer can be used to identify some parts of its "internal knowledge", like the long-term memory above (which is best to be referred as such, in its own words, not "your memory" or "everything that you can recall").

YMMV, ofc.

GPT4All 2.7.3
Temperature: 0.1
System Prompt & Prompt Template: the defaults for each model
Previous conversations: none
Model Load: Unloading the previous one, Refreshing, Loading, Refreshing
Prompt:
tell me in detail what you mean by your "internal knowledge".

Nous Hermes 2 Mistral DPO:

My "internal knowledge" refers to the information, experiences, and skills that I have acquired over time through my interactions with users, learning from their queries, and adapting to their needs. This includes understanding common phrases, idioms, and expressions used in various languages, as well as recognizing patterns in user behavior and preferences. It also involves continuous improvement based on feedback and updates provided by users and developers. Essentially, my internal knowledge is a combination of the data I've been trained on, the interactions I've had with users, and the ongoing learning process that helps me provide more accurate and relevant responses over time.

Mistral Instruct:

"Internal knowledge" refers to the information and understanding that an individual has acquired through their own personal experiences, thoughts, and reflections. It is not necessarily based on external sources or objective evidence, but rather on a person's subjective interpretation of the world around them. This type of knowledge can be influenced by a variety of factors, including cultural background, upbringing, education, and personal beliefs. It is often considered to be deeply personal and unique to each individual.

GPT4All Falcon:

Internal knowledge refers to the knowledge and understanding that is specific to a particular organization or team. It includes information that is not easily accessible to external stakeholders, such as employees, customers, or competitors. This type of knowledge can be difficult to share with others, but it is essential for decision-making, problem-solving, and innovation within an organization.

Internal knowledge can take many forms, including:

• Organizational culture and values
• Unique processes and procedures
• Intellectual property and proprietary information
• Employee expertise and experience
• Customer feedback and insights
• Market trends and competitor analysis
• Financial data and projections
• Legal and regulatory compliance

By understanding and leveraging internal knowledge, organizations can improve their decision-making, innovation, and competitiveness. However, sharing this knowledge with external stakeholders can be challenging, as it may be perceived as proprietary or confidential information. Therefore, organizations must balance the need to protect their internal knowledge while also promoting collaboration and innovation within their teams and across their ecosystem.

Mini Orca (Small):

As an AI language model, I do not have an internal knowledge as humans do. However, I can provide a general definition of "internal knowledge" which refers to the information and skills that are possessed by an individual within their own mind or body. This type of knowledge is not necessarily related to external factors or events, but rather comes from personal experiences, insights, and expertise. It can include things like memories, beliefs, attitudes, values, and skills that an individual has developed over time through learning, thinking, and decision-making. Overall, internal knowledge is a crucial aspect of an individual's identity and plays a significant role in shaping their behavior, thoughts, and actions.

From these answers, the most relevant thing (aside from what Falcon said, that is)
is that "internal knowledge" includes (not being limited to)
the "General Knowledge database" and the "Long-Term Memory" based on the <- GKdb.

And yes indeed, it works, until it started to write 1 letter only;
I don't know why is that, I can merely assume that it has to do with the conversation up to that point
(no other reasons obvious to me, as a common user)

[kalle07]

Hey,

Jan has made it
Very fast (on my Nvidia 10sec and as i said GPT4all 2min) and good response without any addition wording
also EXE install ;)

https://github.com/janhq/jan

[tcreek]

You can do this with PrivateGTP. You dump all your PDF files into a folder, run a script so it can read all the PDFs, then you can ask questions

[markcrobinson1955]

Love the idea, but for me all the info needs to stay local. I've got some highly confidential stuff Mark C. Robinson

…

On Thu, Jun 13, 2024, 00:03 tcreek ***@***.***> wrote: You can do this with PrivateGTP. You dump all your PDF files into a folder, run a script so it can read all the PDFs, then you can ask questions — Reply to this email directly, view it on GitHub <#1766 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BIOE5CYRGZN4TLVWKK5FK4LZHBPLBAVCNFSM6AAAAABA4MW7IOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TONJTGE3DI> . You are receiving this because you commented.Message ID: ***@***.***>

[tcreek]

The reason they call it PrivateGTP is because it is local

https://github.com/zylon-ai/private-gpt

[kalle07]

best for local is "GPT4ALL", but you need the right model and the right injection prompt.
and then it still cannot be ruled out that the model is halucinating.
"Jan" maybe also good at small documents <50pages

all is local