ChatGPT Experiments
It’s been a long time since my last post and many AI related things have happened in the meanwhile. For this post I wanted to get some hands-on experience with ChatGPT’s code generation capabilities. My goal was to build some small utilities using ChatGPT and draw some conclusions.
Basically, I wanted to test scenarios that are fairly common, for instance:
- Dealing with logs, files and directories.
- Using external packages.
- Consuming APIs.
- Storing data in a database.
- Processing a CSV file.
- Using multiple threads.
All the source code that ChatGPT generated is available here and you can find the specifications I gave ChatGPT as text files as well. Keep in mind that ChatGPT keeps improving and changing so you might get a different program if you try my same specifications.
Warning: I’m aware that ChatGPT is much more capable than you’ll see on this post, but my purpose was just to evaluate if I could benefit from it for regular day to day work.
Experiment 1 - A simple backup utility
I like playing Terraria and it’s a good idea to backup your save files in case you do something dumb and ruin your world. Hence, the first program will be a simple backup utility that reads files from a source directory and creates a compressed file in another directory.
It took a lot of experimentation and tweaking, but this is the final spec I provided ChatGPT.
First, I wanted ChatGPT to create a config file:
As for the program, I submitted these instructions:
And here is the program that ChatGPT created. Interestingly, ChatGPT was able to reference the file it created in the previous step and also created code for more complicated logic such as “delete the old zip files in the destination directory but keep the newest zip files based on the value of the number of backups to keep”.
import os
import json
import zipfile
import datetime
import glob
import logging
import sys
def setup_logging(log_file):
# Configure logging to save program activity to log.txt and print to console
logging.basicConfig(
filename=log_file,
level=logging.INFO,
format="%(asctime)s - %(levelname)s: %(message)s",
)
console_handler = logging.StreamHandler(sys.stdout)
console_handler.setLevel(logging.INFO)
formatter = logging.Formatter("%(asctime)s - %(levelname)s: %(message)s")
console_handler.setFormatter(formatter)
logging.getLogger("").addHandler(console_handler)
def create_zip(source_directory, destination_directory):
# Create a zip file
current_datetime = datetime.datetime.now().strftime("%Y%m%d%H%M%S")
zip_file_name = current_datetime + ".zip"
zip_file_path = os.path.join(destination_directory, zip_file_name)
try:
with zipfile.ZipFile(zip_file_path, "w", zipfile.ZIP_DEFLATED) as zipf:
for root, _, files in os.walk(source_directory):
for file in files:
file_path = os.path.join(root, file)
relative_path = os.path.relpath(file_path, source_directory)
zipf.write(file_path, relative_path)
except Exception as e:
logging.error(f"Error creating zip file: {e}")
exit(1)
return zip_file_name
def main():
# Setup logging
log_file = "log.txt"
setup_logging(log_file)
# Log the start of the copy process
logging.info("Copy process started.")
# Read the JSON configuration file
config_file = "backup.json"
if not os.path.exists(config_file):
logging.error(f"Config file not found: {config_file}")
exit(1)
try:
with open(config_file, "r") as json_file:
config = json.load(json_file)
except Exception as e:
logging.error(f"Error reading JSON file: {e}")
exit(1)
source_directory = config.get("sourceDirectory", "")
destination_directory = config.get("destinationDirectory", "")
backups_to_keep = config.get("backupsToKeep", 5)
# Validate source and destination directories
if not os.path.exists(source_directory):
logging.error("Source directory not found")
exit(1)
if not os.path.exists(destination_directory):
logging.error("Destination directory not found")
exit(1)
# Check if source and destination directories are the same
if os.path.abspath(source_directory) == os.path.abspath(destination_directory):
logging.error("Source and destination directories are the same")
exit(1)
zip_file_name = create_zip(source_directory, destination_directory)
# Delete old zip files, keeping the newest zip files based on backups_to_keep
zip_files = glob.glob(os.path.join(destination_directory, "*.zip"))
zip_files.sort(key=os.path.getctime)
for old_zip_file in zip_files[:-backups_to_keep]:
os.remove(old_zip_file)
# Log program activity and print to console
logging.info(f"Source directory: {source_directory}")
logging.info(f"Destination directory: {destination_directory}")
logging.info(f"Zip file created: {zip_file_name}")
logging.info(f"Backups to keep: {backups_to_keep}")
# Log the end of the copy process
logging.info("Copy process finished.")
if __name__ == "__main__":
main()
Let’s test the program:
It works! Just in case, I configured my config file to keep the last 3 files only.
I liked that ChatGPT was able to take care of all the IO boilerplate code. It’s also nice that you can ask ChatGPT to analyze the program it has just created and provide suggestions on how to improve it. Here’s an example:
This experiment took many hours because I was just starting to interact with ChatGPT and I didn’t know what worked and what didn’t work. For sure writing the program myself would have been faster and more fun than writing a detailed specification, but still it was an interesting experiment.
Experiment 2 - Legends of Runeterra Deck Comparer
Legends of Runeterra is a very nice Collectible Card Game. For this experiment the only thing you need to know is that you play with a deck that contains a collection of cards. In order to share decks easily, a deck can be encoded as an alphanumeric string called “deck code”.
For example:
CEBAIAIFB4WDANQIAEAQGDAUDAQSIJZUAIAQCBIFAEAQCBAA
stands for:
What we want to do is compare two decks. Notice that cards have a type (Champions, Units, Spells, etc.). Fortunately, there is a python package that we can leverage in order to parse a deck code.
Again, it took a lot of patience and experimentation, but this is the final spec:
In this case, I had to tweak the spec quite a bit:
- I had to include some modularity in my instructions because the generated code was truly spaghetti code.
- For some reason ChatGPT didn’t understand that the correct method was LoRDeck.from_deckcode and insisted on calling a non-existing method.
However, I especially liked that ChatGPT was able to create the code for a high-level instruction like “diff the results”.
Here is the final program:
import os
import requests
import logging
from lor_deckcodes import LoRDeck
import json
def setup_logger(log_filename):
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] - %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
handlers=[
logging.FileHandler(log_filename),
logging.StreamHandler()
]
)
def download_cards_json(download_folder):
# Check if cards.json exists in the download folder
json_file_path = os.path.join(download_folder, "cards.json")
if os.path.exists(json_file_path):
return
# If the file does not exist, download it
download_url = "https://hextechoracle.com/lor/content/data/cards.json"
response = requests.get(download_url)
if response.status_code == 200:
with open(json_file_path, "wb") as json_file:
json_file.write(response.content)
logging.info("Downloaded cards.json")
else:
logging.error("Failed to download cards.json")
def decode_deck(deck_code, cards_data):
try:
# Decode the deck code using LoRDeck.from_deckcode
deck = LoRDeck.from_deckcode(deck_code)
# Create a list to store card information
cards_list = []
# Collect card information (card_code, count, name, cost, rarity, type) for each card in the deck
for card in deck.cards:
card_count = card.count
card_code = card.card_code
# Get card information from cards_data using card_code
if card_code in cards_data:
card_info = cards_data[card_code]
card_name = card_info["name"]
card_cost = card_info["cost"]
card_rarity = card_info["rarity"]
card_type = card_info["type"]
# Check rarity and type conditions
cards_list.append((card_cost, card_name, card_count, card_rarity, card_type))
else:
logging.warning(f"Card with code {card_code} not found in cards.json")
# Sort the cards_list first by card cost, then by card name
cards_list.sort(key=lambda x: (x[0], x[1]))
return cards_list
except ValueError:
logging.error("Invalid deck code")
return []
def print_deck_contents(deck_code, cards_list):
logging.info(f"Deck Code: {deck_code}")
separator = "-" * 60
# Print cards inside cards_list whose rarity is "Champion"
logging.info("Champion Cards:")
for card in cards_list:
if card[3] == "Champion":
pretty_format = f"Cost: {card[0]:>2}, Name: {card[1]:<30}, Count: {card[2]:>2}"
logging.info(pretty_format)
# Print cards inside cards_list whose type is "Unit" and rarity is not "Champion"
logging.info("Unit Cards (Non-Champion):")
for card in cards_list:
if card[3] != "Champion" and card[4] == "Unit":
pretty_format = f"Cost: {card[0]:>2}, Name: {card[1]:<30}, Count: {card[2]:>2}"
logging.info(pretty_format)
# Print cards inside cards_list whose type is "Spell"
logging.info("Spell Cards:")
for card in cards_list:
if card[4] == "Spell":
pretty_format = f"Cost: {card[0]:>2}, Name: {card[1]:<30}, Count: {card[2]:>2}"
logging.info(pretty_format)
# Print a separator 60 characters long
logging.info(separator)
def load_cards_data(download_folder):
json_file_path = os.path.join(download_folder, "cards.json")
with open(json_file_path, "r", encoding="utf-8") as json_file:
return json.load(json_file)
def main():
# Define the download folder
download_folder = "/workspaces/chatGPT-experiments/deckCompare/download"
# Set up logging
log_filename = "log.txt"
setup_logger(log_filename)
# Download cards.json if needed
download_cards_json(download_folder)
# Load cards.json into a variable
cards_data = load_cards_data(download_folder)
# Deck codes to decode
deck_code1 = "CEBAIAIFB4WDANQIAEAQGDAUDAQSIJZUAIAQCBIFAEAQCBAA"
deck_code2 = "CEBAGAIFB4WDACABAEBQYFAYEESCONACAEAQCBACAECQKNQBAEAQKHI"
# Call the function to decode and print the decks
result1 = decode_deck(deck_code1, cards_data)
result2 = decode_deck(deck_code2, cards_data)
# Print the deck contents with pretty formatting and separator
print_deck_contents(deck_code1, result1)
print_deck_contents(deck_code2, result2)
# Diff the second result versus the first one
diff_results(result1, result2)
def diff_results(result1, result2):
# Find cards missing in result2 compared to result1
missing_in_result2 = [card for card in result1 if card not in result2]
# Find cards missing in result1 compared to result2
missing_in_result1 = [card for card in result2 if card not in result1]
logging.info("Difference between Result 1 and Result 2:")
if missing_in_result1:
logging.info("Cards missing in Result 1:")
for card in missing_in_result1:
pretty_format = f"Cost: {card[0]:>2}, Name: {card[1]:<30}, Count: {card[2]:>2}"
logging.info(pretty_format)
if missing_in_result2:
logging.info("Cards missing in Result 2:")
for card in missing_in_result2:
pretty_format = f"Cost: {card[0]:>2}, Name: {card[1]:<30}, Count: {card[2]:>2}"
logging.info(pretty_format)
if __name__ == "__main__":
main()
Time to show the program in action. Let’s take a look at the decks we are comparing first:
Deck 1 (CEBAIAIFB4WDANQIAEAQGDAUDAQSIJZUAIAQCBIFAEAQCBAA)
Deck 2 (CEBAGAIFB4WDACABAEBQYFAYEESCONACAEAQCBACAECQKNQBAEAQKHI)
From visual inspection we can see that the second deck contains a copy of “Withering Wail” and only two copies of “Grasp of the Undying”. And that’s exactly what our little program reports.
I haven’t tested the program extensively, so if it has some bugs, we’ll just blame ChatGPT. 😃
Experiment 3 - Legends of Runeterra win rate reporter
In case you just skipped to this experiment, take some time to understand what a “Legends of Runeterra deck” is (it’s explained in Experiment 2). Legends of Runeterra records game statistics so it’s possible to know which decks are being played and which ones are having better performance.
In this experiment, we will download win rate data from the excellent Runeterra.ar, store it in a SQLite database and query the results.
This is the program that ChatGPT created:
import os
import requests
import sqlite3
def create_db_and_table():
if not os.path.exists('db'):
os.makedirs('db')
# Create a SQLite database and 'winrate' table
conn = sqlite3.connect('db/winrate.db')
cursor = conn.cursor()
cursor.execute('''
CREATE TABLE IF NOT EXISTS winrate (
name TEXT,
wr REAL,
count INTEGER
)
''')
# Truncate the 'winrate' table
cursor.execute('DELETE FROM winrate')
conn.commit()
conn.close()
def insert_data_into_db(name, wr, count):
conn = sqlite3.connect('db/winrate.db')
cursor = conn.cursor()
cursor.execute('INSERT INTO winrate (name, wr, count) VALUES (?, ?, ?)', (name, wr, count))
conn.commit()
conn.close()
def fetch_and_store_data(url, payload):
try:
response = requests.post(url, json=payload)
if response.status_code == 200:
data = response.json()
meta = data.get("meta", [])
for element in meta:
name = element.get("name", "N/A")
wr = element.get("wr", "N/A")
count = element.get("count", "N/A")
insert_data_into_db(name, wr, count)
else:
print(f"Failed to retrieve data from the URL. Status code: {response.status_code}")
except Exception as e:
print(f"An error occurred: {str(e)}")
def pretty_print_database_records():
conn = sqlite3.connect('db/winrate.db')
cursor = conn.cursor()
cursor.execute('SELECT * FROM winrate')
records = cursor.fetchall()
conn.close()
print("Database Records:")
print("{:<20} {:<10} {:<10}".format("Name", "Win Rate", "Count"))
for record in records:
name, wr, count = record
print("{:<20} {:<10.2f} {:<10}".format(name, wr, count))
def select_top_3_records():
conn = sqlite3.connect('db/winrate.db')
cursor = conn.cursor()
cursor.execute('SELECT * FROM winrate ORDER BY count DESC LIMIT 3')
records = cursor.fetchall()
conn.close()
print("Top 3 Records by Count (Ordered by Count Descending):")
print("{:<20} {:<10} {:<10}".format("Name", "Win Rate", "Count"))
for record in records:
name, wr, count = record
print("{:<20} {:<10.2f} {:<10}".format(name, wr, count))
def main():
# URL to query
url = "https://runeterra.ar/api/meta/get/filter/everyone/en_us?take=20&type=two&filter=true&format=client_Formats_Standard_name&matches=3&wr=3"
# Data to send in the POST request
payload = {
"region": [],
"champ": [],
"set": []
}
create_db_and_table()
try:
# Fetch and store data from the URL
fetch_and_store_data(url, payload)
# Pretty-print database records
pretty_print_database_records()
# Print a separator
print("-" * 50)
# Select and print the top 3 records
select_top_3_records()
except Exception as e:
print(f"An error occurred: {str(e)}")
if __name__ == "__main__":
main()
Let’s test the program:
And… it works as expected.
Maybe it was because I had more experience with ChatGPT at this point, but this spec was easier to write. I particularly liked that you can tell ChatGPT to pretty print values and it generates code that I’ve always found annoying to write.
Experiment 4 - Multiple threads
In this experiment, I wanted ChatGPT to write multithreaded code. Basically, there’s a CSV file with some API URLs inside. We want to call the APIs and to do it faster, we want to distribute the work using threads.
The strategy will be to store the URLs in a queue and have multiple threads steal work from the queue. This is the specification:
And this is the program ChatGPT created:
import aiohttp
import asyncio
import csv
import os
from datetime import datetime
import threading
from queue import Queue
OUTPUT_FOLDER = "output"
async def send_post_request(url, body):
try:
async with aiohttp.ClientSession() as session:
async with session.post(url, data=body) as response:
response_text = await response.text()
return response_text
except Exception as e:
return f"Error sending POST request: {str(e)}"
def process_records(thread_id, queue):
while not queue.empty():
record = queue.get()
url, body = record
response_text = asyncio.run(send_post_request(url, body))
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
response_file = os.path.join(OUTPUT_FOLDER, f"response_{thread_id}.txt")
with open(response_file, "a") as file:
file.write(f"Timestamp: {timestamp}\nURL: {url}\nResponse: {response_text}\n\n")
def main():
csv_file = "/workspaces/chatGPT-experiments/parallel/input.csv"
queue = Queue()
try:
if not os.path.exists(OUTPUT_FOLDER):
os.makedirs(OUTPUT_FOLDER)
else:
for filename in os.listdir(OUTPUT_FOLDER):
if filename.endswith(".txt"):
os.remove(os.path.join(OUTPUT_FOLDER, filename))
with open(csv_file, "r") as csvfile:
csv_reader = csv.reader(csvfile)
header = next(csv_reader) # Read the header row
for row in csv_reader:
if len(row) >= 2:
url, body = row
queue.put((url, body))
else:
print(f"Invalid record in the CSV: {row}")
# Create and start three threads for processing records
threads = []
for i in range(3):
thread = threading.Thread(target=process_records, args=(i + 1, queue))
threads.append(thread)
thread.start()
# Wait for all threads to finish
for thread in threads:
thread.join()
except FileNotFoundError:
print(f"File {csv_file} not found.")
except Exception as e:
print(f"An error occurred: {str(e)}")
if __name__ == "__main__":
main()
As expected, the program created three log files (one per thread)
And we can see that thread #1 processed ids 1 and 5.
In this case, I had never used asyncio before, so it was nice to be able to analyze ChatGPT’s program in order to understand and learn how to use it.
Final thoughts
Based on my experience, these are my conclusions:
- As part of my experiments, I wanted to create 100% correct programs and that took a lot of time and tweaking (in most cases it would have been faster to write the program myself).
- Writing a detailed and correct full specification takes time and it’s boring. However, you can just hack some spec, get an 85% correct program and finish it yourself. This is probably what I would do.
- ChatGPT can save you a lot of time for boring stuff like IO, logging, etc. It can also be quite handy for small functionality like sorting, pretty printing, etc.
- I used Python for this post because programs are usually short and I’m familiar with it, but ChatGPT can be also very useful if you need to learn a new programming language or if you are a beginner.
That’s it! You can find the complete source code here if you are interested.
Thanks for reading!!! 😃