TfidfVectorizer processes first document only - tfidfvectorizer

Question

I'm getting TD-IDF weights only on the 1st document in the list. The rest are zeros!
tfidf_vectorizer = TfidfVectorizer()
X = tfidf_vectorizer.fit_transform(docs)
print(tfidf_vectorizer.idf_)
pd.DataFrame(X[0].T.todense(), index=tfidf_vectorizer.get_feature_names(), columns=["tfidf"]).sort_values(by=["tfidf"],ascending=False).head(10)
For list:
docs=[
"the world dog",
"the cat hello",
"the foo hello",
]
The output
[1.69314718 1.69314718 1.69314718 1.28768207 1. 1.69314718]
tfidf
dog 0.652491
world 0.652491
the 0.385372
cat 0.000000
foo 0.000000
hello 0.000000
After swapping the first two lines
docs=[
"the cat hello",
"the world dog",
"the foo hello",
]
The output becomes:
[1.69314718 1.69314718 1.69314718 1.28768207 1. 1.69314718]
tfidf
cat 0.720333
hello 0.547832
the 0.425441
dog 0.000000
foo 0.000000
world 0.000000
Can anyone offer insight into this issue?

Answer 1

it might be because when you're making a DataFrame you're only using the first value of the TF-IDF's sparse matrix X[0].T.todense() should be X.T.todense()
Also you'd do better if you kept the fitted TFIDF into another variable so you can transform new incoming data for testing:
tfidf_vectorizer = TfidfVectorizer().fit(docs)
X = tfidf_vectorizer.transform(docs)
print(tfidf_vectorizer.idf_)
Now, since you don't have 1 line, you can't transpose and state only 1 column, you'll need to convert to an array an set the columns as the tf-idf possible values. The indes will be numeric.
df = pd.DataFrame(X.toarray(),
columns=tfidf_vectorizer.get_feature_names())
print(df.head(10)) # or display() if you're in jupyter