Ask your files. With Nvidia Chat with RTX, you can start an AI chatbot at home on your computer – Živě.cz

#files #Nvidia #Chat #RTX #start #chatbot #home #computer #Živě.cz

Nvidia has published the demo application Chat with RTX, which already shows the future of working with local files. It uses the existing large Llama or Mistral language models, with which you can insert a local folder with TXT, PDF, DOC/DOCX and XML files or videos and playlists from YouTube.

Using a method called retrieval-augmented generation (RAG), when generating an answer to a query in LLM, the model can rely on real background data and thus avoid hallucinations or ambiguities in the generated text.

And because Nvidia Chat with RTX runs purely locally on local data, it is not only fast, but also secure at the same time. You can thus run queries on protected content that should not be hanging anywhere on the Internet. A lawyer can query laws and past rulings, a technician can consult otherwise impenetrable reference manuals, and a doctor can run queries on patients’ medical records.

The main limitation currently will be primarily the language models used. These are economical to run locally on one computer, and they either do not know Czech at all, or in a very early, almost unusable version. In reality, you can only start it today on the underlying data in English. Even so, it will certainly find wide use, but enthusiasm must be curbed. It is not local ChatGPT 4.

Hardware requirements

You will also need more powerful hardware. A GeForce RTX series 30 or 40 graphics card equipped with at least 8 GB of memory is required. Cheap RTX 4050 or 4060 cards don’t offer a similar amount of graphics memory in basic gaming PCs and laptops, so you need something better. But it’s still an achievable requirement, you don’t have to look for the NVIDIA GH200 Grace Hopper with 288 GB of memory for a nice round million crowns.

Other requirements are already quite achievable – Windows 11, 16 GB of memory and current Nvidia drivers.

Chat with RTX, even in this basic version, shows well the possibilities and limits of local language models. The ability to run on a single computer cut off from the Internet still means concessions in the form of simple answers in a limited number of languages. At the same time, language models require large amounts of memory that cannot be found in ordinary computers.

The demand for performance and occupied memory also indicates that a similar language model on the computer will probably not be permanently ready in the background for a while. As a regular part of the system, we won’t be using it for a few more years, and even adding NPU neural units directly to the processors won’t change that.

But already today, although according to Nvidia it is only a demo application, Chat with RTX can significantly help in many cases when mining information from locally stored documents.