177Lu-PSMA-617 and Pembrolizumab in Treating Patients With Metastatic Castration-Resistant Prostate Cancer

Inclusion Criteria: * The subject is able and willing to comply with study procedures and provide signed and dated informed consent * Histologically confirmed prostate adenocarcinoma. De novo small cell neuroendocrine prostate cancer will not be allowed due to putative lower PSMA expression in this tumor subtype. Treatment-emergent small cell neuroendocrine prostate cancer detected in metastatic tumor biopsy is not an exclusion * A minimum of three PSMA-avid lesions on baseline 68Ga-PSMA-11 PET, with positive lesions defined as those with maximum standardized uptake value (SUVmax) values greater than liver. * Progressive metastatic castration-resistant prostate cancer by Prostate Cancer Working Group (PCWG)3 criteria at the time of study entry * Castrate level of serum testosterone at study entry (\< 50 ng/dL). Participants without prior bilateral orchiectomy are required to remain on luteinizing hormone-releasing hormone (LHRH) analogue treatment for duration of study * Prior progression on at least one second generation androgen signaling inhibitor including abiraterone, apalutamide, darolutamide, and/or enzalutamide * Absolute neutrophil count \> 1.5 x 10\^9/L * Hemoglobin \> 9.0 g/dL * Platelet count \> 100,000/microliter * Serum creatinine =\< 1.5 x upper limit of normal (ULN) or estimated glomerular filtration rate (GFR) \> 50 ml/min by Cockcroft-Gault or 24 hour urine collection * Total bilirubin =\< 1.5 x ULN. In participants with known or suspected Gilbert's disease, direct bilirubin =\< ULN * Aspartate aminotransferase and alanine aminotransferase =\< 2.5 x ULN (\<= 5 x ULN in participants with liver metastases) * No other systemic anti-cancer therapies administered other than LHRH analogue within 14 days, or 5 half-lives, whichever is shorter, prior to initiation of study treatment. Adverse events related to prior anti-cancer treatment other than LHRH analog treatment must have recovered to Grade \<= 1 with the exception of any grade alopecia and grade \<= 2 neuropathy. * Participants must have an Eastern Cooperative Oncology Group (ECOG) performance status of 0 or 1 * Participants must use appropriate methods of contraception during study treatment and for at least 60 days after last study treatment * Participants who are sexually active should consider their female partner to be of childbearing potential if she has experienced menarche and is not postmenopausal (defined as amenorrhea \> 24 consecutive months) or has not undergone successful surgical sterilization. Even women who use contraceptive hormones (oral, implanted, or injected), an intrauterine device, or barrier methods (diaphragms, condoms, spermicide) should be considered to be of childbearing potential * Participants who have undergone vasectomy themselves should also be considered to be of childbearing potential * Acceptable methods of contraception include continuous total abstinence, or double-barrier method of birth control (e.g. condoms used with spermicide, or condoms used with oral contraceptives). Periodic abstinence and withdrawal are not acceptable methods of contraception * Participants must provide consent to comply to recommended radioprotection precautions during study * Participants willing to undergo tumor biopsy and have at least one lesion safely accessible to tumor biopsy. Bone or soft tissue lesion is allowed * Measurable disease by RECIST 1.1 criteria Exclusion Criteria: * Untreated brain metastases at study entry. Participants with previously treated brain metastases are eligible provided the following criteria are all met: * Last treatment was \> 28 days prior to cycle 1 day 1 (C1D1) * No evidence of new/progressive brain metastases is observed on magnetic resonance imaging (MRI) obtained during screening window * Patient is clinically stable without requirement of steroid treatment for at least 14 days prior to first dose of study treatment * Receipt of prior PSMA-directed treatment (e.g. radiotherapy, immunotherapy, or antibody-drug conjugate) * Prior enrollment on clinical study investigating Lu-PSMA-based radioligand therapy * Prior treatment with radium-223 or other radioisotope for the treatment of prostate cancer * Has received prior radiotherapy within 2 weeks of start of study treatment. Participants must have recovered from all radiation-related toxicities, not require corticosteroids, and not have had radiation pneumonitis. A 1-week washout is permitted for palliative radiation (=\< 2 weeks of radiotherapy) to non-central nervous system (CNS) disease * Receipt of prior pembrolizumab or another immune checkpoint inhibitor (e.g. nivolumab, ipilimumab) * Is currently participating in or has participated in a study of an investigational agent or has used an investigational device within 4 weeks prior to the first dose of study treatment * Note: Participants who have entered the follow-up phase of an investigational study may participate as long as it has been 4 weeks after the last dose of the previous investigational agent * Receipt of taxane chemotherapy applied in the castration-resistant setting. Prior receipt of taxane chemotherapy in the hormone-sensitive setting is allowed * Grade \> 2 peripheral neuropathy at the time of study entry * Has severe hypersensitivity (\>= grade 3) to pembrolizumab and/or any of its excipients * Has an active autoimmune disease that has required systemic treatment in the past 2 years (i.e., with use of disease modifying agents, corticosteroids or immunosuppressive drugs). Replacement therapy (e.g., thyroxine, insulin, or physiologic corticosteroid replacement therapy for adrenal or pituitary insufficiency) or treatment with drugs (e.g. neomercazole, carbimazole, etc.) that function to decrease the generation of thyroid hormone by a hyperfunctioning thyroid gland (e.g. in Graves? disease) is not considered a form of systemic treatment of an autoimmune disease * Has a diagnosis of immunodeficiency or is receiving systemic steroid therapy at a prednisone equivalent dose of \> 10 mg daily or other form of immunosuppressive therapy within 7 days prior to first dose of study drug * Has a history of (non-infectious) ≥ grade 2 pneumonitis/interstitial lung disease that required steroids within past 2 years or has current ≥ grade 1 pneumonitis/interstitial lung disease at the time of study enrollment.. * Has received a live vaccine or live-attenuated vaccine within 30 days prior to the first dose of study drug. Administration of killed vaccines is allowed. * Participants who because of age, general medical or psychiatric condition, or physiologic status cannot give valid informed consent * Has clinically significant cardiovascular disease including, but not limited to: * Uncontrolled or any New York Heart Association class 3 or 4 congestive heart failure * Uncontrolled angina, history of myocardial infarction, unstable angina or stroke within 6 months before study entry * Clinically significant arrhythmias not controlled by medication. Chronic rate controlled or paroxysmal atrial fibrillation/flutter is not an exclusion to study participation * Prior external beam radiation involving \>= 25% of bone marrow or within 14 days of start of protocol therapy * Major surgery within 28 days of study treatment \*Note: If participant received major surgery, they must have recovered adequately from the toxicity and/or complications from the intervention prior to starting study treatment. * Has an active infection requiring systemic therapy * Has a known history of human immunodeficiency virus (HIV) (screening not required) * Has a known history of hepatitis B (defined as hepatitis B surface antigen \[HBsAg\] reactive) or known active hepatitis C virus (HCV) (defined as HCV ribonucleic acid \[RNA\] \[qualitative\] is detected) infection (screening not required) * Has a known history of active Bacillus tuberculosis (TB) * Has known psychiatric or substance abuse disorders that would interfere with cooperation with the requirements of the trial * Any condition that, in the opinion of the principal investigator, would impair the patient's ability to comply with study procedures * History of bleeding diathesis and not currently on anti-coagulation therapy that cannot be safely discontinued for the tumor biopsy procedure