fd1857a

Author: Jack-Khuu

llama31-1218/cpu_aoti_4.txt

@@ -0,0 +1,80 @@
+
+OMP_NUM_THREADS=16 numactl --cpunodebind=0 --membind=0 python3 torchchat.py generate llama3.1 --quantize '{"linear:int8": {"groupsize": 0}, "precision": {"dtype":"bfloat16"}, "executor":{"accelerator":"cpu"}}' --prompt "Once upon a time," --max-new-tokens 256 --compile --num-samples 3
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+OMP_NUM_THREADS=16 numactl --cpunodebind=0 --membind=0 python3 torchchat.py generate llama3.1 --quantize '{"linear:int8": {"groupsize": 0}, "precision": {"dtype":"bfloat16"}, "executor":{"accelerator":"cpu"}}' --prompt "Once upon a time," --max-new-tokens 256 --compile --num-samples 3
+PyTorch version 2.6.0.dev20241218+cu124 available.
+Unabled to import torchao experimental quant_api with error:  [Errno 2] No such file or directory: '/home/jackkhuu/oss/torchchat/torchao-build/src/ao/torchao/experimental/quant_api.py'
+Using device=cpu Intel(R) Xeon(R) Platinum 8339HC CPU @ 1.80GHz
+Loading model...
+Time to load model: 0.10 seconds
+Quantizing the model with: {'linear:int8': {'groupsize': 0}, 'precision': {'dtype': 'bfloat16'}, 'executor': {'accelerator': 'cpu'}}
+Time to quantize model: 29.60 seconds
+-----------------------------------------------------------
+Once upon a time, in a small village nestled in the rolling hills of Provence, there lived a young artist named Sophie. Sophie was known throughout the village for her exquisite watercolors of the local flora and fauna. She spent her days painting the vibrant bouquets of wildflowers that bloomed in the nearby fields, and the majestic birds that soared through the sky.
+
+Sophie's studio was a cozy little room above her family's bakery, filled with the sweet scent of freshly baked bread wafting through the air. Her easel stood by the window, where she could paint the ever-changing light of the Provençal sky. Sophie's art was not just a reflection of her love for nature, but also a way for her to connect with the beauty of the world around her.
+
+One day, a wealthy patron from the city, Monsieur LeFleur, arrived in the village in search of a new artist to paint the gardens of his estate. He had heard about Sophie's extraordinary talent and sought her out, eager to commission a series of paintings that would capture the essence of the Provençal landscape.
+
+Sophie was both thrilled and intimidated by the opportunity. She had never painted on such a grand scale before, and the pressure to produce something truly exceptional was daunting.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                
+Generated 255 tokens                 
+Time for inference 1: 176.6127 sec total                 
+Time to first token: 0.6387 sec with parallel prefill.                
+
+      Total throughput: 1.4495 tokens/sec, 0.6899 s/token                 
+First token throughput: 1.5656 tokens/sec, 0.6387 s/token                 
+ Next token throughput: 1.4491 tokens/sec, 0.6901 s/token                     
+
+Bandwidth achieved: 12.41 GB/s
+*** This first iteration will include cold start effects for dynamic import, hardware caches, JIT compilation. ***
+just-in-time compilation time (incl run time): 1.8e+02 seconds
+
+========================================
+
+Once upon a time, there was a young girl named Sophie who lived in a small village surrounded by rolling hills and dense forests. She was a curious and adventurous child, always eager to explore the world around her.
+One day, while wandering through the forest, Sophie stumbled upon a hidden path she had never seen before. The path was overgrown with vines and shrubs, and it looked as though it hadn't been used in years. Sophie's curiosity was piqued, and she decided to investigate further.
+As she made her way down the path, the trees grew taller and the air grew thick with the scent of wildflowers. Sophie felt as though she was walking through a secret world, one that few people knew existed. The path twisted and turned, leading her deeper into the forest.
+Suddenly, Sophie heard the sound of running water. She followed the sound and soon found herself standing at the edge of a beautiful waterfall. The water cascaded down a rocky cliff, creating a misty veil that surrounded her.
+Sophie felt a sense of wonder and awe wash over her. She had never seen anything like it before. She couldn't help but feel a sense of magic in the air, as though the waterfall was a portal to a different world.
+As she stood there, taking in
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                
+Generated 255 tokens                 
+Time for inference 2: 35.0481 sec total                 
+Time to first token: 0.3972 sec with parallel prefill.                
+
+      Total throughput: 7.3043 tokens/sec, 0.1369 s/token                 
+First token throughput: 2.5176 tokens/sec, 0.3972 s/token                 
+ Next token throughput: 7.3591 tokens/sec, 0.1359 s/token                     
+
+Bandwidth achieved: 62.52 GB/s
+
+========================================
+
+Once upon a time, in the rolling hills of Somerset, there was a small village nestled among the picturesque countryside. The village was called Langport, and it was famous for its beautiful riverside setting and its rich history.
+One day, a young girl named Emily moved to Langport with her family. She was excited to explore her new surroundings and make some new friends. As she wandered through the village, she discovered a quaint little shop with a sign that read "Curious Goods".
+Emily's curiosity was piqued, and she pushed open the door to venture inside. The shop was dimly lit, with shelves upon shelves of strange and wondrous objects. There were vintage dolls, antique clocks, and mysterious boxes with strange symbols etched onto their lids.
+The shopkeeper, an elderly man with a kind face and twinkling eyes, welcomed Emily to his store. "Ah, a new face in town," he said with a warm smile. "What brings you to Curious Goods?"
+Emily's eyes widened as she scanned the shelves. "I'm not sure," she said. "I just saw the sign and thought it looked interesting."
+The shopkeeper chuckled. "Ah, curiosity is a wonderful thing. Let me show you some of my favorite treasures."
+As Emily browsed
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                
+Generated 255 tokens                 
+Time for inference 3: 34.1833 sec total                 
+Time to first token: 0.3417 sec with parallel prefill.                
+
+      Total throughput: 7.4890 tokens/sec, 0.1335 s/token                 
+First token throughput: 2.9262 tokens/sec, 0.3417 s/token                 
+ Next token throughput: 7.5351 tokens/sec, 0.1327 s/token                     
+
+Bandwidth achieved: 64.10 GB/s
+
+========================================
+
+
+Warning: Excluding compile in calculations                 
+      Average tokens/sec (total): 7.40                 
+Average tokens/sec (first token): 2.72                 
+Average tokens/sec (next tokens): 7.45 
+                
+Memory used: 0.00 GB

llama31-1218/cpu_aoti_8.txt

@@ -0,0 +1,75 @@
+
+OMP_NUM_THREADS=16 numactl --cpunodebind=0 --membind=0 python3 torchchat.py generate llama3.1 --quantize '{"precision": {"dtype":"bfloat16"}, "executor":{"accelerator":"cpu"}}' --prompt "Once upon a time," --max-new-tokens 256 --compile --num-samples 3
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+OMP_NUM_THREADS=16 numactl --cpunodebind=0 --membind=0 python3 torchchat.py generate llama3.1 --quantize '{"precision": {"dtype":"bfloat16"}, "executor":{"accelerator":"cpu"}}' --prompt "Once upon a time," --max-new-tokens 256 --compile --num-samples 3
+PyTorch version 2.6.0.dev20241218+cu124 available.
+Unabled to import torchao experimental quant_api with error:  [Errno 2] No such file or directory: '/home/jackkhuu/oss/torchchat/torchao-build/src/ao/torchao/experimental/quant_api.py'
+Using device=cpu Intel(R) Xeon(R) Platinum 8339HC CPU @ 1.80GHz
+Loading model...
+Time to load model: 0.11 seconds
+Quantizing the model with: {'precision': {'dtype': 'bfloat16'}, 'executor': {'accelerator': 'cpu'}}
+Time to quantize model: 0.01 seconds
+-----------------------------------------------------------
+Once upon a time, in the ancient kingdom of Aethoria, there lived a young warrior princess named Eira. Eira was known throughout the land for her unmatched bravery, strength, and unwavering dedication to justice. Her people adored her, and her enemies trembled at the mere mention of her name.
+Eira's journey began when she was just a child, training in the ways of combat and magic under the watchful eye of her wise and powerful mentor, the sorceress Arachne. Arachne had taken Eira under her wing, recognizing the young princess's innate potential and grooming her to one day take the throne.
+As Eira grew in power and wisdom, she became increasingly frustrated with the injustices that plagued her kingdom. Corruption, tyranny, and oppression were rampant, and Eira saw it as her duty to put an end to them. She spent countless hours studying the ancient lore and magical texts, seeking a deeper understanding of the world and the forces that shaped it.
+One fateful day, a dark and malevolent force began to spread across the land, threatening to destroy everything Eira held dear. A powerful sorcerer-king, named Malakar, had risen to power, using his mastery of dark magic to enslave the people
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                
+Generated 255 tokens                 
+Time for inference 1: 220.2756 sec total                 
+Time to first token: 0.6765 sec with parallel prefill.                
+
+      Total throughput: 1.1622 tokens/sec, 0.8605 s/token                 
+First token throughput: 1.4782 tokens/sec, 0.6765 s/token                 
+ Next token throughput: 1.1612 tokens/sec, 0.8612 s/token                     
+
+Bandwidth achieved: 18.67 GB/s
+*** This first iteration will include cold start effects for dynamic import, hardware caches, JIT compilation. ***
+just-in-time compilation time (incl run time): 2.2e+02 seconds
+
+========================================
+
+Once upon a time, there lived a rabbit named Rosie. Rosie was different from the other rabbits in the forest. While they were content to spend their days nibbling on carrots and lounging in the sun, Rosie had a passion for adventure.
+Rosie had heard of a hidden garden deep within the forest, full of the most beautiful and exotic flowers she had ever seen. She had always dreamed of exploring it, but she was afraid of the unknown dangers that lay within.
+One day, Rosie decided that she had had enough of being afraid. She packed a small bag with some carrots, a canteen of water, and a map, and set off on her journey to the hidden garden.
+As she wandered through the forest, Rosie encountered all sorts of obstacles. She had to navigate through thick underbrush, cross rushing streams, and climb steep hills. But she persevered, driven by her determination to reach the garden.
+Finally, after what seemed like hours of walking, Rosie caught sight of a glimpse of green through the trees. She quickened her pace, her heart racing with excitement.
+As she pushed through the final curtain of foliage, Rosie gasped in wonder. Before her lay a garden unlike any she had ever seen. The flowers were more vibrant and exotic than she had imagined,
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                
+Generated 255 tokens                 
+Time for inference 2: 81.8519 sec total                 
+Time to first token: 0.4276 sec with parallel prefill.                
+
+      Total throughput: 3.1276 tokens/sec, 0.3197 s/token                 
+First token throughput: 2.3387 tokens/sec, 0.4276 s/token                 
+ Next token throughput: 3.1317 tokens/sec, 0.3193 s/token                     
+
+Bandwidth achieved: 50.23 GB/s
+
+========================================
+
+Once upon a time, in a land far, far away, there was the most magical city in all the land. It was a place where dreams came true and magic was real. The city was called Everwood, and it was a place of wonder and enchantment.
+Everwood was a city of towering trees that seemed to stretch up to the sky, their branches tangled and woven together in a way that seemed almost... magical. The trees were covered in leaves that shimmered and sparkled in the sunlight, and the air was filled with the sweet scent of honey and lavender.
+In the heart of Everwood, there was a great and ancient tree, the Heartwood. It was said that the Heartwood was the source of all magic in the city, and that it held the secrets of the ancient ones. The Heartwood was a place of great power and wisdom, and many sought to unlock its secrets.
+In Everwood, the inhabitants were a magical people, with the ability to communicate with animals and control the elements. They lived in harmony with the natural world, and their city was a reflection of this harmony. Everwood was a place of beauty and wonder, where anything seemed possible.
+But, as magical as Everwood was, it was not without its dangers. There were dark
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                
+Generated 255 tokens                 
+Time for inference 3: 83.6049 sec total                 
+Time to first token: 0.4247 sec with parallel prefill.                
+
+      Total throughput: 3.0620 tokens/sec, 0.3266 s/token                 
+First token throughput: 2.3546 tokens/sec, 0.4247 s/token                 
+ Next token throughput: 3.0656 tokens/sec, 0.3262 s/token                     
+
+Bandwidth achieved: 49.18 GB/s
+
+========================================
+
+
+Warning: Excluding compile in calculations                 
+      Average tokens/sec (total): 3.09                 
+Average tokens/sec (first token): 2.35                 
+Average tokens/sec (next tokens): 3.10 
+                
+Memory used: 0.00 GB