On-Premises Llama 2 및 SingleStoreDB 를 이용한 생성형 AI 프로그램 예제

$ git clone https://github.com/ggerganov/llama.cpp $ cd llama.cpp $ git reset --hard d0cee0d36d5be95a0d9088b674dbb27354107221 $ curl -L "https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGUF/resolve/main/llama-2-7b-chat.Q4_K_S.gguf" -o models/llama-2-7b-chat.Q4_K_S.gguf $ sudo yum install -y gcc-c++ git $ sudo yum install gcc-toolset-12 $ scl enable gcc-toolset-12 bash $ make

$ ./main -m ./models/llama-2-7b-chat.Q4_K_S.gguf -p "[INST] <<SYS>>\nYou are a helpful assistant.\n<</SYS>>\nTell me about the relations between Typhoon Khanun and Jamboree in South Korea[/INST]" -n 512 main: build = 1087 (d0cee0d36) main: seed = 1775538028 ... llm_load_print_meta: format = GGUF V2 (latest) llm_load_print_meta: arch = llama llm_load_print_meta: vocab type = SPM llm_load_print_meta: n_vocab = 32000 llm_load_print_meta: n_merges = 0 llm_load_print_meta: n_ctx_train = 4096 llm_load_print_meta: n_ctx = 512 llm_load_print_meta: n_embd = 4096 llm_load_print_meta: n_head = 32 llm_load_print_meta: n_head_kv = 32 llm_load_print_meta: n_layer = 32 llm_load_print_meta: n_rot = 128 llm_load_print_meta: n_gqa = 1 llm_load_print_meta: f_norm_eps = 1.0e-05 llm_load_print_meta: f_norm_rms_eps = 1.0e-06 llm_load_print_meta: n_ff = 11008 llm_load_print_meta: freq_base = 10000.0 llm_load_print_meta: freq_scale = 1 llm_load_print_meta: model type = 7B llm_load_print_meta: model ftype = mostly Q4_K - Small llm_load_print_meta: model size = 6.74 B llm_load_print_meta: general.name = LLaMA v2 llm_load_print_meta: BOS token = 1 '<s>' llm_load_print_meta: EOS token = 2 '</s>' llm_load_print_meta: UNK token = 0 '<unk>' llm_load_print_meta: LF token = 13 '<0x0A>' llm_load_tensors: ggml ctx size = 0.09 MB llm_load_tensors: mem required = 3677.46 MB (+ 256.00 MB per state) .................................................................................................. llama_new_context_with_model: kv self size = 256.00 MB llama_new_context_with_model: compute buffer total size = 71.91 MB system_info: n_threads = 4 / 4 | AVX = 1 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | sampling: repeat_last_n = 64, repeat_penalty = 1.100000, presence_penalty = 0.000000, frequency_penalty = 0.000000, top_k = 40, tfs_z = 1.000000, top_p = 0.950000, typical_p = 1.000000, temp = 0.800000, mirostat = 0, mirostat_lr = 0.100000, mirostat_ent = 5.000000 generate: n_ctx = 512, n_batch = 512, n_predict = 512, n_keep = 0 [INST] <<SYS>>\nYou are a helpful assistant.\n<</SYS>>\nTell me about the relations between Typhoon Khanun and Jamboree in South Korea[/INST] I'm happy to help! However, I must inform you that there is no relation between Typhoon Khanun and Jamboree in South Korea. Our records show no information regarding the two weather phenomena or their possible interactions. Typhoon Khanun is a tropical cyclone that typically affects the Pacific region, particularly the Philippines and Taiwan. It is important to note that typhoons can have severe impacts on these regions, including strong winds, heavy rainfall, and storm surge. Jamboree, on the other hand, is not a weather phenomenon but rather a type of event. Jamboree is a large-scale gathering or celebration that brings together people from various backgrounds to share experiences, knowledge, and culture. While jamborees can be fun and exciting events, they are unrelated to weather phenomena like typhoons. I hope this information helps clarify any confusion! If you have any other questions, please feel free to ask. [end of text] llama_print_timings: load time = 430.55 ms llama_print_timings: sample time = 132.19 ms / 216 runs ( 0.61 ms per token, 1634.01 tokens per second) llama_print_timings: prompt eval time = 5765.72 ms / 48 tokens ( 120.12 ms per token, 8.33 tokens per second) llama_print_timings: eval time = 32379.64 ms / 215 runs ( 150.60 ms per token, 6.64 tokens per second) llama_print_timings: total time = 38326.87 ms

$ python -m venv .venv $ source .venv/bin/activate $ pip install llama-cpp-python $ pip install langchain-community $ pip install langchain-text-splitters $ pip install langchain-singlestore $ pip install langchain-classic $ pip install sentence-transformers

import os from langchain_community.document_loaders import DirectoryLoader, TextLoader from langchain_community.embeddings import HuggingFaceEmbeddings from langchain_text_splitters import RecursiveCharacterTextSplitter from langchain_singlestore.vectorstores import SingleStoreVectorStore embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2", model_kwargs={'device': 'cpu'}) loader = DirectoryLoader('./', glob='khanun.txt', loader_cls=TextLoader) documents = loader.load() text_splitter = RecursiveCharacterTextSplitter(chunk_size=500,chunk_overlap=50) texts = text_splitter.split_documents(documents) os.environ["SINGLESTOREDB_URL"] = "user:pass@127.0.0.1:3306/vectordb" vectorstore = SingleStoreVectorStore.from_documents(texts, embeddings, distance_strategy="DOT_PRODUCT", table_name="demo")

import os from langchain_community.embeddings import HuggingFaceEmbeddings from langchain_community.llms import LlamaCpp from langchain_singlestore.vectorstores import SingleStoreVectorStore from langchain_classic.chains import RetrievalQA from langchain_core.prompts import PromptTemplate embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2", model_kwargs={'device': 'cpu'}) os.environ["SINGLESTOREDB_URL"] = "user:pass@127.0.0.1:3306/vectordb" vectorstore = SingleStoreVectorStore(embeddings, distance_strategy="DOT_PRODUCT", table_name="demo") qa_template = """[INST] <<SYS>> Use the context provided to answer the user's question. Be direct and concise. Do not say you don't know if the answer is in the context. <</SYS>> Context: {context} Question: {question} [/INST] """ prompt = PromptTemplate(template=qa_template, input_variables=['context', 'question'])

llm = LlamaCpp(model_path="/home/opc/llama.cpp/models/llama-2-7b-chat.Q4_K_S.gguf", n_gpu_layers=1, n_batch=512, n_ctx=2048, f16_kv=True, verbose=False) dbqa = RetrievalQA.from_chain_type(llm=llm, chain_type='stuff', retriever=vectorstore.as_retriever(search_kwargs={'k': 5}), return_source_documents=True, chain_type_kwargs={'prompt': prompt})

response = dbqa.invoke({'query':'Tell me about the relations between Typhoon Khanun and Jamboree in South Korea'}) print(f'\nAnswer: {response["result"]}') Answer: In response to the user's question, Typhoon Khanun had an impact on the 25th World Scout Jamboree in South Korea. Due to the changed path and expected impact of the typhoon, the Korean government, in cooperation with the World Organization of the Scout Movement, decided to end the ongoing jamboree early. All attendees will be relocated to hotels and accommodation in Seoul or whatever the contingent has decided on.